1 2011-09-12T00:18:48 <dreimark> moin
2 2011-09-12T00:38:53 <ThomasWaldmann> somehow the new stuff is much easier
3 2011-09-12T00:39:06 <ThomasWaldmann> at least yet
4 2011-09-12T00:48:08 <dreimark> hopefully no new problems
5 2011-09-12T01:09:22 <ThomasWaldmann> existing tests have currently no failures
6 2011-09-12T01:36:01 <dreimark> gn
7 2011-09-12T01:41:39 <ronny> ThomasWaldmann: already integrating storage-ng in a branch?
8 2011-09-12T01:44:09 <ThomasWaldmann> no, i am integrating whoosh indexing in storage-ng
9 2011-09-12T01:44:34 <ronny> i see, anything pushed?
10 2011-09-12T01:44:54 <ThomasWaldmann> indexing not yet, that is quite complex and much
11 2011-09-12T01:45:17 <ThomasWaldmann> but if you want to help, you can fork the repo and improve the tests as you proposed
12 2011-09-12T01:46:32 <ThomasWaldmann> or in general have a look whether you find stuff to improve
13 2011-09-12T01:47:14 <ronny> k
14 2011-09-12T01:49:23 *** Marchael
15 2011-09-12T01:50:40 <ronny> ThomasWaldmann: i'll have to do tommorow, not enough concentration
16 2011-09-12T01:51:16 <ThomasWaldmann> it's already tomorrow :D
17 2011-09-12T01:52:12 <ronny> ok, after sleep
18 2011-09-12T04:49:11 *** Marchael
19 2011-09-12T05:36:16 *** Marchael
20 2011-09-12T09:21:34 *** pkumar
21 2011-09-12T11:06:19 *** pkumar
22 2011-09-12T11:33:01 <ThomasWaldmann> moin
23 2011-09-12T11:50:40 <ronny> moin
24 2011-09-12T12:00:48 <ThomasWaldmann> btw, indexing hat erste lebenszeichen
25 2011-09-12T12:03:26 <ThomasWaldmann> and i noted that building the latest revs index is not trivial / not cheap
26 2011-09-12T12:21:11 *** greg_f
27 2011-09-12T12:21:31 *** mkerrin
28 2011-09-12T12:21:54 *** greg_f
29 2011-09-12T12:23:10 <ronny> ThomasWaldmann: what exactly is the issue?
30 2011-09-12T12:26:49 <ThomasWaldmann> i think i have an algo:
31 2011-09-12T12:27:01 <ThomasWaldmann> # TODO: build ix_latest using ix_all, algorithm idea:
32 2011-09-12T12:27:02 <ThomasWaldmann> # first determine set(revids) - set(parent_revids) (we need both in index)
33 2011-09-12T12:27:04 <ThomasWaldmann> # ---> these are all heads of all items
34 2011-09-12T12:27:07 <ThomasWaldmann> # now group by itemid, sort groups by mtime (reverse)
35 2011-09-12T12:27:09 <ThomasWaldmann> # each first rev in a run of same-item revs is the latest revision
36 2011-09-12T12:27:12 <ThomasWaldmann> # index latest revisions to ix_latest
37 2011-09-12T12:30:42 <ronny> will that happen before or afer indexing all revisions?
38 2011-09-12T12:31:04 <ThomasWaldmann> after. "using ix_all"
39 2011-09-12T12:31:37 <ThomasWaldmann> before indexing everything, we have no clue about stuff anyway.
40 2011-09-12T12:32:19 <ThomasWaldmann> (this is for the index rebuild, when we have data, but lost index)
41 2011-09-12T12:34:43 <ronny> ThomasWaldmann: can woosh return sorted/grouped results?
42 2011-09-12T12:35:10 <ThomasWaldmann> yes (in principle, have to check if we can directly use that)
43 2011-09-12T12:36:15 <ronny> ThomasWaldmann: i looks we can sort by uuid, mtime
44 2011-09-12T12:37:14 <ThomasWaldmann> hmm, right, that might be easier, no set building
45 2011-09-12T12:37:15 <ronny> ThomasWaldmann: http://packages.python.org/Whoosh/facets.html looks promising
46 2011-09-12T12:39:44 <ThomasWaldmann> yup, just a bit confusing
47 2011-09-12T12:40:33 <ronny> yah
48 2011-09-12T12:40:49 <ronny> ThomasWaldmann: from what i understand we need to group by uuid, mtime
49 2011-09-12T12:41:48 <ronny> the question is if there is a way to always select the max
50 2011-09-12T12:52:17 <ThomasWaldmann> we'll see :)
51 2011-09-12T12:52:30 <ThomasWaldmann> i currently prepare committing the indexing stuff so you can see it
52 2011-09-12T12:55:27 <ronny> ok
53 2011-09-12T13:00:01 <ThomasWaldmann> ok, i committed/pushed
54 2011-09-12T13:03:14 <ThomasWaldmann> i took the opportunity to cleanup the existing indexing stuff also
55 2011-09-12T13:03:24 <ThomasWaldmann> Marchael: ^^
56 2011-09-12T13:03:57 <ThomasWaldmann> https://bitbucket.org/thomaswaldmann/storage-ng/changesets
57 2011-09-12T13:04:59 <ThomasWaldmann> no routing yet, btw
58 2011-09-12T13:05:16 <Marchael> moin
59 2011-09-12T13:06:12 <Marchael> ThomasWaldmann: that means rewriting a lot of indexing code?
60 2011-09-12T13:06:39 <Marchael> or just some changes in index generation algorithm?
61 2011-09-12T13:07:03 <Marchael> s/generation/building/
62 2011-09-12T13:07:04 <ThomasWaldmann> no, we did rather fundamental changes about how stuff works, about how code is arranged
63 2011-09-12T13:07:53 <ThomasWaldmann> but stuff calling storage.documents(...) or .search(...) will still work
64 2011-09-12T13:09:04 <ThomasWaldmann> i just wasn't to happy about the storage api and that each of my tries to incrementally improve it failed
65 2011-09-12T13:09:31 <Marchael> so we need to adapt whoosh code in index script to new backend changes, am I right?
66 2011-09-12T13:09:35 <ThomasWaldmann> so ronny and me redesigned it
67 2011-09-12T13:10:07 <ThomasWaldmann> currently that is too early, the stuff is still isolated in a separate repo
68 2011-09-12T13:10:43 <ThomasWaldmann> we try to make it work there as good as possible and then transplant the code
69 2011-09-12T13:11:04 <ThomasWaldmann> after that, yes, changes to the moin index script will be required
70 2011-09-12T13:11:29 <ThomasWaldmann> you can have a look at the code at the url i gave, it is all much easier now
71 2011-09-12T13:11:44 <ThomasWaldmann> not all is implemented yet, though
72 2011-09-12T13:12:18 <ThomasWaldmann> we'll reuse quite some of your code, but not all :)
73 2011-09-12T13:13:13 <ThomasWaldmann> but it is nice to see how whoosh made stuff simple
74 2011-09-12T13:15:11 <Marchael> hm, is that something I could do? may be add sorting by some criterias to search form?
75 2011-09-12T13:16:51 <ThomasWaldmann> if you like to help, review the new code whether you find something suspect
76 2011-09-12T13:17:41 <ThomasWaldmann> or fork the repo if you want to do code changes
77 2011-09-12T13:18:13 <Marchael> ok
78 2011-09-12T13:18:25 * ThomasWaldmann will try to bring that stuff into a good state asap, so it can be transplanted
79 2011-09-12T13:21:48 <ThomasWaldmann> ronny: did you change something in EP recently?
80 2011-09-12T13:33:51 <ronny> ThomasWaldmann: not that i know of, is something wrong?
81 2011-09-12T13:51:12 <ThomasWaldmann> no, i just want to clean up there
82 2011-09-12T13:54:45 <ronny> k
83 2011-09-12T13:54:54 <ronny> im afk for a bit, im not really well yet
84 2011-09-12T14:11:31 *** madhadron
85 2011-09-12T14:13:12 *** madhadron
86 2011-09-12T14:32:02 <dreimark> moin
87 2011-09-12T14:53:17 <ThomasWaldmann> hi dreimark
88 2011-09-12T16:11:19 <ronny> ThomasWaldmann: ping?
89 2011-09-12T16:11:42 <ThomasWaldmann> yes?
90 2011-09-12T16:13:28 <ThomasWaldmann> I'll change that rebuild() stuff to internally use an iterator over revids (one for all revs, one for latest revs) later and reduce duplication that way.
91 2011-09-12T16:13:39 <ThomasWaldmann> ronny: ?
92 2011-09-12T16:14:50 <ronny> ThomasWaldmann: im back
93 2011-09-12T16:16:29 <ronny> ThomasWaldmann: i think revisions need a different api, creation shouldnt go via the wrapper object
94 2011-09-12T16:18:53 <ThomasWaldmann> keep router and names in mind
95 2011-09-12T16:19:49 <ronny> ThomasWaldmann: item.create_revision(meta, data) should do
96 2011-09-12T16:20:01 <ronny> i changed the pad a bit
97 2011-09-12T16:20:29 <ronny> revision objects should only be applicable to existing revisions
98 2011-09-12T16:20:43 <ronny> creation should be the concern of item
99 2011-09-12T16:31:18 * ThomasWaldmann looks
100 2011-09-12T16:39:46 *** Marchael
101 2011-09-12T16:45:26 *** greg_f
102 2011-09-12T16:55:27 *** ronny
103 2011-09-12T16:55:37 *** ronny
104 2011-09-12T16:59:33 <ThomasWaldmann> ronny: see push
105 2011-09-12T17:04:36 *** brunomartin
106 2011-09-12T17:12:55 *** mkerrin
107 2011-09-12T17:21:24 <ronny> ThomasWaldmann: i get a strange fail in indexing
108 2011-09-12T17:22:47 <ThomasWaldmann> ronny: it has no stuff in latest-revs index yet, see TODO in rebuild
109 2011-09-12T17:23:05 * ThomasWaldmann pushed and must go now for a while
110 2011-09-12T17:23:26 <ronny> oh, i see
111 2011-09-12T17:41:16 * ThomasWaldmann tries to fix the rebuild now
112 2011-09-12T17:43:06 <ronny> ThomasWaldmann: im surprised by woosh querying, its so strange
113 2011-09-12T17:43:19 <ThomasWaldmann> what's strange?
114 2011-09-12T17:43:46 <ronny> i tried to figure how to make it just give me the documents i want in the order i want, it always needs some kind of query object
115 2011-09-12T17:43:57 <ronny> and from the docs i couldnt figure where to get that in a few minutes
116 2011-09-12T17:44:27 <ThomasWaldmann> Term()
117 2011-09-12T17:45:06 <ThomasWaldmann> except if you have a query string, then you need a query parser that will construct the query for you.
118 2011-09-12T17:46:46 <ronny> Term() fails actually
119 2011-09-12T17:47:48 <ronny> hmm, basically i want it to get me all documents sorted somehow
120 2011-09-12T17:51:43 <ThomasWaldmann> there's a special Everything term
121 2011-09-12T17:51:53 <ThomasWaldmann> are you working on same stuff as I do? :)
122 2011-09-12T17:52:25 <ronny> playing with it for understanding
123 2011-09-12T17:52:45 <ThomasWaldmann> (I am still refactoring rebuild to have less duplication, but after that i need a latest_revids iterator
124 2011-09-12T17:52:48 <ThomasWaldmann> )
125 2011-09-12T17:53:49 <ronny> i'll figure a optimal query
126 2011-09-12T17:56:42 * ThomasWaldmann pushed
127 2011-09-12T17:57:12 <ThomasWaldmann> ok, I'll do some non-python stuff now, maybe pull before continuing
128 2011-09-12T17:57:25 <ThomasWaldmann> ronny: ^^
129 2011-09-12T17:58:06 <ThomasWaldmann> all we need is that latest_revids iterator, then it should work
130 2011-09-12T17:59:30 <ronny> ThomasWaldmann: i'll see how to make it work via search
131 2011-09-12T18:03:32 <ThomasWaldmann> I'ld expect it to be something like searcher.search(Everything(), sortedby=ITEMID, limit=None, ...)
132 2011-09-12T18:04:37 <ThomasWaldmann> I'ld expect it to be something like searcher.search(Everything(), sortedby=[ITEMID, MTIME], limit=None, ...)
133 2011-09-12T18:05:07 <ThomasWaldmann> I'ld expect it to be something like searcher.search(Everything(), sortedby=[ITEMID, MTIME], reverse=True, limit=None, ...)
134 2011-09-12T18:06:50 <ThomasWaldmann> then yield first revid of each new ITEMID run and throw away all next hits until the ITEMID changes
135 2011-09-12T18:07:14 *** Marchael
136 2011-09-12T18:12:03 <ronny> ThomasWaldmann: thats roughlt what im doing
137 2011-09-12T18:12:12 <ronny> ThomasWaldmann: i wonder if there is a per group limit
138 2011-09-12T18:17:02 <ronny> ThomasWaldmann: but for now, itertools.groupby rocks
139 2011-09-12T18:17:04 <ThomasWaldmann> that way you do not need groups
140 2011-09-12T18:17:52 <ThomasWaldmann> but it maybe could be useful to have it grouped, so one directly can just take the first entry of each group
141 2011-09-12T18:18:13 <ThomasWaldmann> i didn't use groups yet, not sure what's simpler
142 2011-09-12T18:19:27 <ronny> ThomasWaldmann: it doesnt return groups in the resultset here tho
143 2011-09-12T18:23:42 <ronny> ThomasWaldmann: it returns document id's for the groupings tho
144 2011-09-12T18:35:10 <ThomasWaldmann> if the docs aren't right, file bugs, author is quite responsive
145 2011-09-12T18:38:39 <ronny> hmm
146 2011-09-12T18:38:44 <ronny> ThomasWaldmann: figured stuff
147 2011-09-12T18:38:51 <ronny> ThomasWaldmann: we have a stupid unhelpfull bug
148 2011-09-12T18:38:58 <ronny> we close files we send in
149 2011-09-12T18:39:11 <ronny> which means if revision caches the data it gets from the creation
150 2011-09-12T18:39:14 <ronny> it breaks
151 2011-09-12T18:39:30 <ronny> since closed stringio drops the buffer
152 2011-09-12T18:48:58 <ronny> ok, got it
153 2011-09-12T18:50:58 <ronny> ThomasWaldmann: btw, what is mtime by default?
154 2011-09-12T18:53:05 <ronny> ThomasWaldmann: i pushed to my fork
155 2011-09-12T18:53:31 <ronny> https://bitbucket.org/RonnyPfannschmidt/storage-ng/overview
156 2011-09-12T18:53:44 <ronny> i have the impression something is wrong
157 2011-09-12T18:54:13 <ronny> we need real content checks i think
158 2011-09-12T19:00:11 <ThomasWaldmann> ronny: mtime key is not there currently
159 2011-09-12T19:00:38 <ThomasWaldmann> but you could inject the current time at the same place where it sets the size and hash
160 2011-09-12T19:01:28 * ThomasWaldmann just fetched a new ip to get rid of a kiddie trying to abuse my sshd here
161 2011-09-12T19:02:04 <ronny> ThomasWaldmann: dont you have some kind of attack block scripts for failed logins?
162 2011-09-12T19:02:34 <ThomasWaldmann> you can't login here with password, thus i usually don't need that
163 2011-09-12T19:03:18 <ThomasWaldmann> https://bitbucket.org/RonnyPfannschmidt/storage-ng/changeset/78457b466d18 comments like in 87 aren't that helpful if you don't tell there what's the problem
164 2011-09-12T19:04:44 <ronny> ThomasWaldmann: fixed
165 2011-09-12T19:05:52 <ronny> ThomasWaldmann: what do we do wrt closed stringio's
166 2011-09-12T19:06:15 <ronny> ThomasWaldmann: i think for the memory backend we should pretty much combine getvalue + storing strings internally
167 2011-09-12T19:07:05 <ThomasWaldmann> the closing of files has to be reviewed and document how it is right
168 2011-09-12T19:07:29 <ThomasWaldmann> i mean "who is responsible for closing and when he has to do it"
169 2011-09-12T19:07:46 <ThomasWaldmann> then we maybe get away with a seek(0)
170 2011-09-12T19:07:57 <ronny> ThomasWaldmann: basically its a big to pass data into revision when doing create_revision
171 2011-09-12T19:08:05 <ronny> s/big/bug/
172 2011-09-12T19:08:38 <ThomasWaldmann> big bug? :)
173 2011-09-12T19:08:42 <ThomasWaldmann> why?
174 2011-09-12T19:08:55 * ThomasWaldmann .oO(starship troopers)
175 2011-09-12T19:09:29 <ronny> its a regex :P
176 2011-09-12T19:09:49 * ThomasWaldmann just joking
177 2011-09-12T19:09:51 <ronny> ThomasWaldmann: basically we should always fetch data from the backend after insertion
178 2011-09-12T19:11:15 <ronny> (meta is a mapping that gets mutated, data should always be a file object the backend returned)
179 2011-09-12T19:11:50 <ronny> sounds ok?
180 2011-09-12T19:12:01 <ThomasWaldmann> i can't follow you, give more details
181 2011-09-12T19:12:49 <ThomasWaldmann> the spacing in test_rebuild looks strange
182 2011-09-12T19:13:45 <ThomasWaldmann> and what about line 147, line 113 in test_indexing.py?
183 2011-09-12T19:15:01 <ronny> mom, i think i missed a comparisation
184 2011-09-12T19:19:01 <ronny> ThomasWaldmann: the issue with data files is, that stores close the data sources
185 2011-09-12T19:20:31 <ronny> brb, erand
186 2011-09-12T19:22:26 *** brunomartin
187 2011-09-12T19:24:56 *** raignarok
188 2011-09-12T19:29:18 <ThomasWaldmann> ronny: the rebuild test still fails
189 2011-09-12T19:30:57 <ThomasWaldmann> latest rebuilt index doesn't have the "bar" revision with mtime 2
190 2011-09-12T19:50:23 *** brunomartin
191 2011-09-12T20:33:37 <ThomasWaldmann> ronny: https://bitbucket.org/thomaswaldmann/storage-ng/changeset/d5d311be369e
192 2011-09-12T20:33:55 <ThomasWaldmann> hopefully fixes "close" issues
193 2011-09-12T20:34:34 <ThomasWaldmann> brunomartin: https://bitbucket.org/thomaswaldmann/storage-ng ronny and i fix the storage issues in a radical way there :)
194 2011-09-12T20:35:56 <brunomartin> :-) I will take a look!
195 2011-09-12T20:35:58 <ronny> re
196 2011-09-12T20:36:32 <ThomasWaldmann> ronny: if that solves the "stringio sucks" issue, you may remove the comment again
197 2011-09-12T20:39:18 <ronny> ThomasWaldmann: looks reasonable at first glance
198 2011-09-12T20:39:33 <ronny> ThomasWaldmann: it wil still break if we put in strings like streams
199 2011-09-12T20:39:44 <ronny> s/strings/things/
200 2011-09-12T20:39:50 <ronny> since sreams arent seekable
201 2011-09-12T20:40:09 <ronny> i think its a right step tho
202 2011-09-12T20:40:28 <ronny> but revision.data shouldNEVER come from the input, and always come from the output
203 2011-09-12T20:41:05 <ThomasWaldmann> file / line?
204 2011-09-12T20:42:52 <ronny> indexing.py last line of code
205 2011-09-12T20:43:07 <ronny> we should never assign data from what came in
206 2011-09-12T20:43:26 <ThomasWaldmann> that's exactly what we write into backend
207 2011-09-12T20:43:45 <ThomasWaldmann> ("content" is only computed for index, not for backend)
208 2011-09-12T20:44:35 <ronny> ThomasWaldmann: yes, but if you actually try to get content from data, its a input filedesriptor
209 2011-09-12T20:44:49 <ronny> ThomasWaldmann: which may be a consumed non-seekable stream
210 2011-09-12T20:45:11 <ronny> and end up in index corruption or a error
211 2011-09-12T20:45:29 <ronny> well, not physical corruption, bu certainly incorrect data
212 2011-09-12T20:45:32 <ThomasWaldmann> give and example for a non-seekable stream we need there
213 2011-09-12T20:46:18 <ThomasWaldmann> (if we have that, we have 2 issues, because we need 2 seeks :)
214 2011-09-12T20:46:56 <ronny> hmm, im not taking into account again, that werkzeug makes big uploads to tmpfiles
215 2011-09-12T20:47:29 <ronny> but we might need a smarter wiki serializer/deserializer
216 2011-09-12T20:47:51 <ronny> btw, does it have to be xml for the serialization/deserialization?
217 2011-09-12T20:48:37 <ThomasWaldmann> it is not a requirement, but it worked so far
218 2011-09-12T20:49:06 <ronny> the new model seems very fit for actually writing a tarfile
219 2011-09-12T20:49:18 <ThomasWaldmann> no tarfiles
220 2011-09-12T20:49:26 <ThomasWaldmann> and no zip either
221 2011-09-12T20:50:09 <ronny> imagine serializing backends in a way that would allow tar -x to get a working wiki fs storage
222 2011-09-12T20:50:45 <ThomasWaldmann> that can't work in general as backends are different
223 2011-09-12T20:51:54 <ronny> thin the end serialize/deserialize will be a stream of metadata objects + data objects
224 2011-09-12T20:52:15 <ronny> -th
225 2011-09-12T20:52:28 <ThomasWaldmann> yes, it got much easier now
226 2011-09-12T20:52:53 <ronny> i think its a good idea to serialize into a tarball that would be roughly a valid fs backend store
227 2011-09-12T20:53:20 <ronny> from what i understoo its pretty much only a issue of naming the files so it would end up in such a way
228 2011-09-12T20:53:35 <ThomasWaldmann> i still think it should be not coupled, should be independant of each other
229 2011-09-12T20:54:40 <ronny> it wont look like a usual tarball
230 2011-09-12T20:54:45 <ThomasWaldmann> because otherwise if we change the backend, we automatically get a mismatch in the serialization and need to change there also
231 2011-09-12T20:55:35 <ronny> the idea is that serializing and deserializing uses a format, that can untar to a fs backend
232 2011-09-12T20:55:51 <ThomasWaldmann> also, when deserialising likely most of the time is needed for index building, so we have an issue, but at another place
233 2011-09-12T20:55:55 <ronny> but there would still be need for deserializing
234 2011-09-12T20:56:35 <ThomasWaldmann> yes, and if we change fs backend, it won't have that "can untar to fs backend" property any more
235 2011-09-12T20:56:35 <ronny> the idea is just that a tarball that only needs a index rebuild is better than something complex
236 2011-09-12T20:57:34 <ronny> ok, lets still go for tar as format tho
237 2011-09-12T20:57:42 <ThomasWaldmann> if you use a fs backend, you can create that tarball at any time with tar :D
238 2011-09-12T20:58:55 <ThomasWaldmann> and as that serialization format is also made for migrations, maybe xml or json is more generic than having people build tar files
239 2011-09-12T20:59:20 <ThomasWaldmann> (which is ok on python, but maybe not in their favourite language)
240 2011-09-12T21:00:39 <ronny> ThomasWaldmann: the thing is we have metadata, which is json, and data, which is either text or blobs
241 2011-09-12T21:01:21 <ronny> so some kind of demited binary makes sense
242 2011-09-12T21:01:21 <ThomasWaldmann> metadata is a dict, not necessarily json :)
243 2011-09-12T21:01:39 <ronny> it makes sense to limit to json
244 2011-09-12T21:01:48 <ronny> instant generic/portable
245 2011-09-12T21:02:24 <ThomasWaldmann> if the json deserializer can do what we need, we could use that
246 2011-09-12T21:02:56 <ThomasWaldmann> but big binary data could be a problem, my xml serializer/deserializer works nicely with that
247 2011-09-12T21:03:41 <ronny> hmm
248 2011-09-12T21:03:54 <ronny> i ont understand what ordering whoosh sorting asumes by default
249 2011-09-12T21:04:00 <ThomasWaldmann> but we likely could do chunking with json also, somehow
250 2011-09-12T21:04:13 <ThomasWaldmann> by default it is not reverse
251 2011-09-12T21:04:54 <ThomasWaldmann> result = searcher.search(Every(), groupedby=ITEMID, sortedby=[MTIME], reverse=True)
252 2011-09-12T21:05:04 <ThomasWaldmann> don't you need to give limit=None ?
253 2011-09-12T21:05:31 <ThomasWaldmann> and if sortedby is only one value, you don't need a list iirc
254 2011-09-12T21:06:48 <ThomasWaldmann> search(q, limit=10, sortedby=None, reverse=False, groupedby=None, optimize=True, filter=None, mask=None, groupids=True, terms=False)
255 2011-09-12T21:08:49 <ronny> ThomasWaldmann: im using groupedby, if i reverse, the order is wrong, if i dont, its right
256 2011-09-12T21:08:53 <ronny> its strange
257 2011-09-12T21:09:01 <ThomasWaldmann> huh?
258 2011-09-12T21:09:40 <ThomasWaldmann> did you try without []?
259 2011-09-12T21:10:10 * ThomasWaldmann brb
260 2011-09-12T21:21:46 <ronny> hum
261 2011-09-12T21:21:48 <ronny> heisenbug
262 2011-09-12T21:25:04 <ThomasWaldmann> i guess we need close method and context manager in Revision
263 2011-09-12T21:26:59 *** waldi
264 2011-09-12T21:27:01 *** franklin
265 2011-09-12T21:27:02 *** jek
266 2011-09-12T21:27:04 *** waldi
267 2011-09-12T21:27:04 *** waldi
268 2011-09-12T21:27:07 *** franklin
269 2011-09-12T21:27:13 *** jek
270 2011-09-12T21:43:02 *** xorAxAx
271 2011-09-12T21:43:42 *** xorAxAx
272 2011-09-12T21:55:20 <ronny> ThomasWaldmann: every 3rd-4th test run i get a test fail with the indexer
273 2011-09-12T21:55:28 <ronny> ThomasWaldmann: so something is messed up
274 2011-09-12T21:56:27 <ThomasWaldmann> traceback?
275 2011-09-12T21:57:31 <ThomasWaldmann> pushed contextmanager for Revision
276 2011-09-12T21:58:52 <ronny> ThomasWaldmann: no traceback, sometimes the order is just different
277 2011-09-12T21:59:09 <ThomasWaldmann> ronny: and i guess you are right, we should not patch the input file into the revision, but rather open the file in the storage
278 2011-09-12T21:59:44 <ThomasWaldmann> bit less efficient maybe, but safer
279 2011-09-12T22:01:41 <ThomasWaldmann> ehrm
280 2011-09-12T22:01:48 <ThomasWaldmann> i have NO failures any more :D
281 2011-09-12T22:02:42 <ronny> oh, huh
282 2011-09-12T22:02:46 <ronny> ThomasWaldmann: run it a few times
283 2011-09-12T22:05:34 <ThomasWaldmann> pushed
284 2011-09-12T22:06:04 <ThomasWaldmann> yes, sometimes 1 failure
285 2011-09-12T22:14:22 <ThomasWaldmann> # stringio sucks
286 2011-09-12T22:14:22 <ThomasWaldmann> # once closed anywhere, the buffers are dropped this kills the in-memory backends
287 2011-09-12T22:14:30 <ThomasWaldmann> do we still have that problem?
288 2011-09-12T22:21:27 <ronny> ThomasWaldmann: something about group ordering in whoosh
289 2011-09-12T22:23:03 <ThomasWaldmann> how is that related to that comment?
290 2011-09-12T22:24:42 <ronny> ThomasWaldmann: thats related to sometimes 1 error
291 2011-09-12T22:24:47 <ronny> the comment can go now i guess
292 2011-09-12T22:26:25 <ThomasWaldmann> ok, i remove it
293 2011-09-12T22:34:52 <dreimark> re
294 2011-09-12T22:40:29 * ThomasWaldmann pushed MTIME and CONTENT improvements
295 2011-09-12T22:43:19 * dreimark has 1 failing test
296 2011-09-12T22:43:33 <dreimark> middleware/_tests/test_indexing.py ......F..
297 2011-09-12T22:44:03 <dreimark> http://paste.pocoo.org/show/474898/
298 2011-09-12T22:45:44 <dreimark> if it fails it don't have neccessarily the same data in the traceback
299 2011-09-12T22:46:28 <dreimark> bar can be before foo
300 2011-09-12T22:48:11 <ThomasWaldmann> yes, that one sometimes fails
301 2011-09-12T22:48:20 <dreimark> watch py.test, result toggles between fail and no fail
302 2011-09-12T22:48:35 <ThomasWaldmann> i guess we have an issue with order
303 2011-09-12T22:49:57 <ThomasWaldmann> dreimark: did you follow the confluence converter?
304 2011-09-12T22:50:51 <dreimark> not recently, always if there was something new on the wiki
305 2011-09-12T22:51:27 <dreimark> https://gitorious.org/confluence2moinmoin
306 2011-09-12T22:51:34 <dreimark> last change in august
307 2011-09-12T22:52:23 <ThomasWaldmann> i don't find anything related to moin there
308 2011-09-12T22:54:55 <ronny> ThomasWaldmann: fixed the issue
309 2011-09-12T22:55:00 <ronny> and killed some code in the fs backend
310 2011-09-12T22:55:15 <ronny> since shutil has copyfileob(src, dest, blocksize)
311 2011-09-12T22:56:33 <dreimark> ThomasWaldmann: http://moinmo.in/ConfluenceConverter was linked from there
312 2011-09-12T22:58:23 <ThomasWaldmann> ronny: did you try sortedby=MTIME?
313 2011-09-12T22:58:44 <ronny> ThomasWaldmann: yes
314 2011-09-12T22:58:57 <ronny> ThomasWaldmann: for a strange reason the order is not always stable
315 2011-09-12T22:59:04 <ronny> i havent figured the reason
316 2011-09-12T22:59:51 <ThomasWaldmann> can you file a bug?
317 2011-09-12T23:00:05 <ThomasWaldmann> bitbucket.org/mchaput
318 2011-09-12T23:06:49 <ronny> ThomasWaldmann: im not sure what to report
319 2011-09-12T23:07:14 <ThomasWaldmann> just point him at the code you wrote and why you wrote it
320 2011-09-12T23:11:45 <ronny> i reported https://bitbucket.org/mchaput/whoosh/issue/197/sort-order-with-grouping-sometimes-not
321 2011-09-12T23:20:46 <ThomasWaldmann> ok
322 2011-09-12T23:33:37 *** raignarok
323 2011-09-12T23:52:58 *** brunomartin
324
