[?] MiloDB: Finding Old Teases

[FrozenWolf]

The changes are great and seem to be working well for me in my testing thus far. Thank you!

That's nice to know, thanks.

It took me a bit to figure out how to use the NOT operator, but I think I did. Hint: it goes early in the statement. Here's a complicated one I made:

Code: Select all

query title matches mistress|princess and tags contains denial and not text contains cock and rating > 3.0 and rating <= 4.0

That's great! I'll add an example to the help that includes '

not

' so there's a point of reference in the future.

I don't know if it would be difficult to add, but for me it might be nice to be able to interchangeably/legally use "tag" or "tags".

That's a very interesting point. I've definitely been neglecting tags as you know! I can certainly add '

tag

' as a synonym for '

tags

' without any problems. At the moment I allow '

tags is something

' which doesn't sound right and is fairly meaningless as the tags are stored as, e.g. ':kneeling:female-top:sister:outdoors:for-men:'; and it would only match against the whole thing (which is fairly unlikely). In a future version, I'll be treating tags as a proper list, more like the text, so the verbs will operate against each separate tag. With that in mind, I might just change the field name to '

tag

' so that '

tag is kneeling

', '

tag contains kneel

', '

tag matches fem.+top

' makes sense. The advantage of the word '

text

' is that it's a mass noun, so it sounds correct in all contexts. I'm still not sure, maybe I'll support both '

tag

' and '

tags

' as you suggest and then if you find that you're only using one of them, I'll get rid of the other one.

<date suggestions>
Above are my suggestions, but I don't think that level of comparison is terribly necessary. I don't see any reason why using a full date would be that much more difficult. Adding the shorthand would just be a "nice to have" type thing.

Thanks, that's just what I was after. Maybe I'll start with the longhand version and add the shorthand version if it's relatively trivial.

[FrozenWolf]

0.7, 2022-03-17, beta

• Fixed: Sorting results by rating is now consistent
• Fixed: Teases with no rating showed a rating of "0" instead of "0.0"
• Fixed: Rogue characters in titles and summaries are now stripped
• Fixed: Tags are now shown as a list of items
• Added: Query field 'date' supports verbs '>', '<', '>=', '<=', '='
• Added: Tags can now be searched as separate items
• Added: Field 'tag' is now a synonym for 'tags'

Notes
I've implemented the date comparison using the long-format for now, i.e.:

> query date >= 2021-01-01 and date < 2021-07-01 and title contains day

Tags are significantly improved and now treated as a list.

• '

  tag

  ' supports verbs '

  contains

  ', '

  matches

  ', '

  is

  '
• '

  tags

  ' supports verbs '

  contains

  ', '

  matches

  '

They both do the same thing, but I didn't like the grammar of '

tags is female-top

' so it's not supported . I expect that the name 'tags' will eventually be dropped altogether.

Future
There are still some good things to work on that have been suggested, but anything else you can think of (or if I've missed something), please let me know.

Bugs

• Bug0004: [M] Provide list of allowed values on error from 'type is'
• Bug0005: [M] Add help notes of allowed values for 'type is'

Requests

• Req0004: [L] Enable column sorting in generated web pages
• Req0005: [L] Add field to identify TOTM nominees and winners
• Req0006: [L] Links from the generated web pages could open a new tab or window
• Req0008: [M] Add help example for using NOT in a query
• Req0010: [L] Query by date could support shorthand dates, e.g. '>= 2020'
• Req0011: [L] Provide TAB hints for 'type is' values

[Trusfrated]

Another fantastic update!

Other than what has already said, there's not much that I would change. The more you use the keywords, the more they become natural and it becomes easier to create good queries. There are a couple of things that I still had to correct myself from doing, but that doesn't mean that they should be changed:

• I have a tendency to eliminate the spaces on either side of the comparison operators (<,>,=, and so on). For me it is more natural to not have that extra space (especially next to the numbers), for others it may be normal and easier to read with them. A very minor thing.
• I assumed the date comparison would use my Windows regional settings to determine how I prefer to write the long form of dates. The program expects yyyy-mm-dd, while I prefer mm/dd/yyyy. Again, this is very minor and it's no problem to conform. I just thought that it could be written different ways.

Here's my latest:

Code: Select all

query date < 2016-01-01 and type is nyx and rating >= 4.0 and tag is "verbal-humiliation" and tag contains chastity and text contains bitch and not text contains slut

This is a really nice tool! (That's what she said! )

[FrozenWolf]

have a tendency to eliminate the spaces on either side of the comparison operators (<,>,=, and so on). For me it is more natural to not have that extra space (especially next to the numbers), for others it may be normal and easier to read with them. A very minor thing.

This was a strong consideration, and I decided to do take the simple (easy) approach . The tiny lexer/tokeniser that I wrote isn't context aware, and therefore doesn't know what to expect next, so it only splits on spaces and parenthesis at the moment. That's one of the reasons that if you have a group in a regular expression, it needs to be quoted, e.g.

Code: Select all

> query text matches chastity.?(belt|cage)

is split into:

Code: Select all

'query', 'text', 'matches', 'chastity.?', '(', 'belt|cage', ')'
   1       2         3           4         5        6        7

That produces an error on token #5 because it's expecting an operator such as 'and', or 'or'. A context aware lexer would be aware that it was expecting a regular expression and that a space is the expected delimiter, so it wouldn't split at the '('. It would be a similar issue if the '=', '<', '>' symbols appeared anywhere within a token.

It's certainly worth another look and I'll put it on the list, but I'll consider it as low priority for now as it's quite a lot of work. I hope the error message is good enough for now.

I assumed the date comparison would use my Windows regional settings to determine how I prefer to write the long form of dates. The program expects yyyy-mm-dd, while I prefer mm/dd/yyyy. Again, this is very minor and it's no problem to conform. I just thought that it could be written different ways.

Date and time formatting tends to strike fear and dread into the hearts of software developers . At least there is no timezone and daylight savings time to worry about here! Python does seem to have some support for parsing locale specific date formats, so I think I might be able to allow the following:

Code: Select all

> query date > 2021-08-17      Universal as it conforms to the ISO-8601 date format
> query date > 8/17/2021       US English locale
> query date > 17/9/21         UK English locale

Using the wrong format in the wrong locale will cause an error.

The range of date variants across the world is wonderfully diverse (based on a quick web search, this is not an exhaustive list and may not be completely correct, so apologies in advance). I'm not convinced that all of them have 4-digit years either.

Code: Select all

14.11.2014      Bulgarian
14.11.2014      Czech
14-11-2014      Danish
14.11.2014      German
14/11/2014      Greek
11/14/2014      English (US)
14/11/14        English (UK)
14/11/2014      Spanish
14.11.2014      Estonian
14.11.2014      Finnish
14/11/2014      French
14.11.2014.     Croatian
2014.11.14.     Hungarian
14/11/2014      Italian
2014.11.14      Lithuanian
2014.11.14.     Latvian
14-11-2014      Dutch
14.11.2014      Norwegian
2014-11-14      Polish
14/11/2014      Portuguese
14.11.2014      Romanian
14. 11. 2014    Slovak
14.11.2014      Slovenian
2014-11-14      Swedish
14.11.2014      Turkish
2014/11/14      Chinese

So the approach I'd use would be to try and parse against the ISO-8601 format, and if that fails, try a locale specific parse to see if that works. That's going to be tough for me to test though as I only have access to en_GB and en_US and can only hope that it works elsewhere. As before, I'll add it to the list as low priority.

Here's my latest

Nice one, that's the kind of thing that I like to see!

[FrozenWolf]

TOTM Question
Do you think this should be another field that works a little bit like '

type

', e.g.

Code: Select all

> query totm is winner
> query totm is nominee
> query totm is none

It could also be added an an implicit tag (if the tag isn't already there):

Code: Select all

> query tag is totm-winner
> query tag is totm-nominee
> query not (tag is totm-winner or tag is totm-nominee)

There might be other ways of doing it too. Any thoughts?

[Trusfrated]

TOTM Question
Do you think this should be another field that works a little bit like '

type

', e.g.

Code: Select all

> query totm is winner
> query totm is nominee
> query totm is none

It could also be added an an implicit tag (if the tag isn't already there):

Code: Select all

> query tag is totm-winner
> query tag is totm-nominee
> query not (tag is totm-winner or tag is totm-nominee)

There might be other ways of doing it too. Any thoughts?

That's a good question.

First, I would note that the "winner tag" is actually "totm". There are two teases that used "totm-winner" but that was the author's choice. One actually won (so it has both tags) and the other was a remake of the winning tease.

What would be easier for you? Given that there are already many teases in the database that successfully scraped those tags, my inclination would just be to "correct" the tag list (based on the badges). This seems like it would be cleaner, rather than having two fields that have duplicate data for some. But I can also imagine that populating a new field with this data might be easier to begin with.

Your first example (querying a totm field) is easier for the user, I think. But I don't think there should be partially duplicated data in two fields. So either it's a new field and the tag field is "cleaned" of the partial tags, or the tag list is appended with the proper award tags. If the tag field is used, then perhaps querying a "virtual" totm field (looking at the tag list) could be a later development.

Sorry if I sound indecisive. I can really see it both ways, too.

[bobrosssub]

Wow, I love this :O
Can I have access to the beta? There's some teases I remember vividly but have no idea how to find

[FrozenWolf]

First, I would note that the "winner tag" is actually "totm". There are two teases that used "totm-winner" but that was the author's choice. One actually won (so it has both tags) and the other was a remake of the winning tease.

Good spot, thanks.

What would be easier for you? Given that there are already many teases in the database that successfully scraped those tags, my inclination would just be to "correct" the tag list (based on the badges). This seems like it would be cleaner, rather than having two fields that have duplicate data for some. But I can also imagine that populating a new field with this data might be easier to begin with.

There's not much in it difficulty wise. The framework is reasonably generic now so adding new fields for the queries isn't too hard.

Your first example (querying a totm field) is easier for the user, I think. But I don't think there should be partially duplicated data in two fields. So either it's a new field and the tag field is "cleaned" of the partial tags, or the tag list is appended with the proper award tags. If the tag field is used, then perhaps querying a "virtual" totm field (looking at the tag list) could be a later development.

Let's go with the tags approach for now then by adding '

totm

' and '

totm-nominee

' if they're not already there. There's no problem changing it later if necessary.

Wow, I love this :O
Can I have access to the beta? There's some teases I remember vividly but have no idea how to find

Sure thing, I'll send a PM to you.

[FrozenWolf]

0.8, 2022-03-22, beta

• Fixed: Tags for TOTM nominees and winners are now patched into the database
• Fixed: Errors from missing or invalid 'type is' parameters now show a list of options
• Added: Sort by TOTM is now possible with 'totm' sort key
• Added: Help for query now includes example for using 'not' operator
• Added: Webpages produced by 'browse' and 'browseall' have an icon and a better title
• Added: Webpages produced by 'browse' and 'browseall' show a TOTM icon for relevant teases
• Added: Queries for 'type is' has TAB support for suggestion and completion
• Added: Windows - Executable now has an icon
• Changed: Removed 'tags' as a queryable field
• Changed: Improved formatting of help output
• Changed: Displayed list of expected possible inputs on error is now sorted

Notes
I've re-scraped the entire metadata and the TOTM tags are now patched into the database for all teases. The database itself now only runs up to the end of 2021 because that's the intention for the public release at the end of this month. Teases that have since disappeared have been purged from the database, and those that were updated during 2021 have been updated in the database.

Future
I'm not aware of any bugs at the moment, but there are bound to be some in there. This release could be the first release candidate and therefore could be last beta if there aren't any significant issues found.

Bugs

• None (famous last words)

Requests

• Req0004: [L] Enable column sorting in generated web pages
• Req0006: [L] Links from the generated web pages could open a new tab or window
• Req0010: [L] Query by date could support shorthand dates, e.g. '>= 2020'
• Reg0012: [L] Don't require spaces around the '>', '<', '>=', '<=', '=' verbs
• Req0013: [L] Query by date could support local specific dates, e.g. '>= 5/14/2020' in en_US

[Trusfrated]

0.8, 2022-03-22, beta

First, nice new icons!

Having the TOTM data in there is really significant, I think. So glad you were able to get that! I still think it was a good call to make them tags. This might seem to be contradicting my earlier statement about putting them in a new field, but I wish there was an indicator for them in the List view. These are basically "reserved tags" and are special, so putting an additional character maybe between the rating and the date to designate if the tease contains either of these tags would be nice. I think this would make more sense especially when you sort (and List) on TOTM.

This is kind of like what you have done on the Browse page with the medals. This looks really nice. (Having the indicator by the Title versus the rating is fine; I could see it either way). I don't know if you made those graphics yourself or if they are "stock" but having the "1st" and "2nd" text in there bugged me slightly. I might've gone with no text, just the gold and silver, since there are usually several nominees and I don't know that they would all necessarily be considered "2nd place" awards. (I know this is nit-picky.)

What do you think about saving preferences? I noticed that my setting for Ellipsis doesn't stick between sessions. It seems like Sort and Format preferences would be nice to carry between sessions, too.

I'm anxious for more people to try this!

[FrozenWolf]

First, nice new icons!

Thanks, it took me far too long to craft those!

Having the TOTM data in there is really significant, I think. So glad you were able to get that! I still think it was a good call to make them tags. This might seem to be contradicting my earlier statement about putting them in a new field, but I wish there was an indicator for them in the List view.

These are basically "reserved tags" and are special, so putting an additional character maybe between the rating and the date to designate if the tease contains either of these tags would be nice. I think this would make more sense especially when you sort (and List) on TOTM.

I was struggling a little to find the best way to incorporate them into the list view that didn't take too much space but was still obvious as to what it meant. I did experiment with '

TOTM

', '

TOTM-Nominee

', '

TOTM-W

', '

TOTM-N

', '

Win

', '

Nom

' as a column but those looked terrible when there weren't any in the list and there was just a large blank column. Prefixing the title with '

[TOTM]

', '

[TOTM-Nominee]

', '

[TOTM-W]

', '

[TOTM-N]

', '

[Win]

', '

[Nom]

' looked too much like that was part of the title, and still took too much room.

Did you have an idea of what the character would be if I did put it between the rating and date? I considered 'W' and 'N' (or 'w' and 'n') but wasn't sure if it would be obvious enough what it meant.

This is kind of like what you have done on the Browse page with the medals. This looks really nice. (Having the indicator by the Title versus the rating is fine; I could see it either way). I don't know if you made those graphics yourself or if they are "stock" but having the "1st" and "2nd" text in there bugged me slightly. I might've gone with no text, just the gold and silver, since there are usually several nominees and I don't know that they would all necessarily be considered "2nd place" awards. (I know this is nit-picky.)

I tried initially with the scaled down TOTM icons from the Milovana website but they looked like a red splat and it was impossible to distinguish between the winners and nominees. The medal icons are hand crafted SVG so it's easy for me to remove the '1st' and '2nd'; I'll do that.

What do you think about saving preferences? I noticed that my setting for Ellipsis doesn't stick between sessions. It seems like Sort and Format preferences would be nice to carry between sessions, too.

I did consider updating the config file with those settings (or using an alternative config file for session settings), but I found that I didn't use the tool often enough to warrant remembering those settings, and actually preferred if going back to the defaults. One of the advantages of a GUI would be that those settings could always be visible through check-boxes or drop-down lists, rather than being a bit more hidden with the CLI. If I have the chance to do a GUI, those settings will almost certainly be maintained between sessions.

[Trusfrated]

Having the TOTM data in there is really significant, I think. So glad you were able to get that! I still think it was a good call to make them tags. This might seem to be contradicting my earlier statement about putting them in a new field, but I wish there was an indicator for them in the List view.

These are basically "reserved tags" and are special, so putting an additional character maybe between the rating and the date to designate if the tease contains either of these tags would be nice. I think this would make more sense especially when you sort (and List) on TOTM.

I was struggling a little to find the best way to incorporate them into the list view that didn't take too much space but was still obvious as to what it meant. I did experiment with '

TOTM

', '

TOTM-Nominee

', '

TOTM-W

', '

TOTM-N

', '

Win

', '

Nom

' as a column but those looked terrible when there weren't any in the list and there was just a large blank column. Prefixing the title with '

[TOTM]

', '

[TOTM-Nominee]

', '

[TOTM-W]

', '

[TOTM-N]

', '

[Win]

', '

[Nom]

' looked too much like that was part of the title, and still took too much room.

Did you have an idea of what the character would be if I did put it between the rating and date? I considered 'W' and 'N' (or 'w' and 'n') but wasn't sure if it would be obvious enough what it meant.

Yeah, I agree that the list in nice and concise and you wouldn't want to clutter it up.
Some ideas:

Code: Select all

Winner                Nominee
X                      O
*                      #
W                      N
♥                      ♠

I kind of lean toward the idea of an ansi graphic like two of the card suits, although I don't know if the two I chose make any more sense than the others. Of course, to keep the columns straight you would need to insert a blank space where no award was given. If you want it to be completely obvious what they mean, maybe it's not such a good idea. It was just a thought.

[Carnal1]

I've been following this thread waiting for the release. Does sound very handy. One thing I might suggest if possible, is to not delete from your database teases that have been deleted from Milovana but instead mark them as no longer available or something. More than a few times I have searched for some vaguely remembered tease fruitlesly only to eventually conclude it had probably been deleted. If I knew it was deleted I could try alternate sources.

[FrozenWolf]

Spoiler: show

Having the TOTM data in there is really significant, I think. So glad you were able to get that! I still think it was a good call to make them tags. This might seem to be contradicting my earlier statement about putting them in a new field, but I wish there was an indicator for them in the List view.

These are basically "reserved tags" and are special, so putting an additional character maybe between the rating and the date to designate if the tease contains either of these tags would be nice. I think this would make more sense especially when you sort (and List) on TOTM.

I was struggling a little to find the best way to incorporate them into the list view that didn't take too much space but was still obvious as to what it meant. I did experiment with '

TOTM

', '

TOTM-Nominee

', '

TOTM-W

', '

TOTM-N

', '

Win

', '

Nom

' as a column but those looked terrible when there weren't any in the list and there was just a large blank column. Prefixing the title with '

[TOTM]

', '

[TOTM-Nominee]

', '

[TOTM-W]

', '

[TOTM-N]

', '

[Win]

', '

[Nom]

' looked too much like that was part of the title, and still took too much room.

Did you have an idea of what the character would be if I did put it between the rating and date? I considered 'W' and 'N' (or 'w' and 'n') but wasn't sure if it would be obvious enough what it meant.

Yeah, I agree that the list in nice and concise and you wouldn't want to clutter it up.
Some ideas:

Code: Select all

Winner                Nominee
X                      O
*                      #
W                      N
♥                      ♠

I kind of lean toward the idea of an ansi graphic like two of the card suits, although I don't know if the two I chose make any more sense than the others. Of course, to keep the columns straight you would need to insert a blank space where no award was given. If you want it to be completely obvious what they mean, maybe it's not such a good idea. It was just a thought.

It's definitely still a good idea!

The two card suits are, unfortunately, not portable as they're ASCII characters chr(3) and chr(6) for ♥ and ♠ respectively. In this case, they work fine under a Windows terminal with codepage 850/437 but they're consumed under Linux as control characters and may also not appear under other Windows codepages/locales. It's definitely a tricky one, but having some sort of indicator is definitely a good idea.

This is what some of the options look like:

Code: Select all

  1:  #18024  reg  4.3  w  2012-09-10  Tonights the Night : [hornbound, @23267]
  2:  #19533  nyx  3.9  n  2017-09-26  Before tonight you will be tested... : [T€ch, @25441]
  3:  #30861  reg  3.9     2015-08-14  Tonight's the night? : [Cruel Ally, @35975]
  4:  #52952  reg  3.7     2021-10-08  Honey, what are we gonna do tonight? : [Parrezo25, @111878]

Code: Select all

  1:  #18024  reg  4.3  W  2012-09-10  Tonights the Night : [hornbound, @23267]
  2:  #19533  nyx  3.9  N  2017-09-26  Before tonight you will be tested... : [T€ch, @25441]
  3:  #30861  reg  3.9     2015-08-14  Tonight's the night? : [Cruel Ally, @35975]
  4:  #52952  reg  3.7     2021-10-08  Honey, what are we gonna do tonight? : [Parrezo25, @111878]

Code: Select all

  1:  #18024  reg  4.3  *  2012-09-10  Tonights the Night : [hornbound, @23267]
  2:  #19533  nyx  3.9  #  2017-09-26  Before tonight you will be tested... : [T€ch, @25441]
  3:  #30861  reg  3.9     2015-08-14  Tonight's the night? : [Cruel Ally, @35975]
  4:  #52952  reg  3.7     2021-10-08  Honey, what are we gonna do tonight? : [Parrezo25, @111878]

Code: Select all

  1:  #18024  reg  4.3  X  2012-09-10  Tonights the Night : [hornbound, @23267]
  2:  #19533  nyx  3.9  O  2017-09-26  Before tonight you will be tested... : [T€ch, @25441]
  3:  #30861  reg  3.9     2015-08-14  Tonight's the night? : [Cruel Ally, @35975]
  4:  #52952  reg  3.7     2021-10-08  Honey, what are we gonna do tonight? : [Parrezo25, @111878]

I'm going to use 'w' and 'n' for now.

When there are no indicators:

Code: Select all

  1:    #988  reg  4.0     2007-08-24  Anniversary Tease : [slaveboy00, @71]
  2:   #1970  reg  3.5     2008-02-12  The Humiliatrix Sisters Use And Abuse You Part 1: Penelope : [cerebus, @1614]
  3:   #2266  reg  4.1     2008-03-23  Eighth Shemale or Not Quiz : [MontSpam, @2184]
  4:   #2444  reg  4.0     2008-05-06  Caught and Trained : [spice23, @881]

I think that looks OK.

[FrozenWolf]

I've been following this thread waiting for the release. Does sound very handy. One thing I might suggest if possible, is to not delete from your database teases that have been deleted from Milovana but instead mark them as no longer available or something. More than a few times I have searched for some vaguely remembered tease fruitlesly only to eventually conclude it had probably been deleted. If I knew it was deleted I could try alternate sources.

It's certainly possible for me to keep the existing tease content in the database and, as you suggest, mark it as deleted. The last web scrape was a special case where I scanned the whole set of teases from the beginning of time and therefore could see what had disappeared, but in future I'll only be looking at one month at time (from three months in the past), so I won't capture any tease that's been removed within three months of it being published, and I won't be checking historical teases beyond three months to see if they're still there.

I will keep it in mind though, there might be some opportunities in the future.