Milovana Size Limitation

[indyc]

Trials of the Succubi has been too large to edit normally for about 2 years now. Through fapnip and other's help, I was able to find ways to edit it in multiple teases to merge them together. However, ever since TOTS 3.0, they have been too large to officially publish on this site even with some serious compression efforts.

In the last 2 months or so, I have been having trouble uploading the JSON after editing the tease in sperate parts. Apparently the size limitation for uploading a JSON for a "new tease" is larger than overriding/updating JSON of an existing tease. I wouldn't be surprised if I'm an upload or two away from being unable to upload my updated JSON regardless.

A while back seraph0x made this post:

I'm curious if you know how I can publish tease number :
https://milovana.com/webteases/showteas ... c0a6f57297
or even better, the larger:
https://milovana.com/webteases/showteas ... 1c1945e848

In the past they would both have a timeout error on publishing but now it is giving me a 500 error.

Adding support for bigger teases is definitely high on the priority list but I want to get through some of these bug fixes first. Honestly, when I originally built Eos, I would have never dreamt that folks would want to build these insanely complex experiences. Now that I know there is a demand for that, I can definitely add some optimizations that make that experience better but it'll require some fairly fundamental changes to the engine.

If any of these efforts could be looked at again it would be a HUGE game changer for me. I know it is not a simple request but I think it would also affect large teases like Legend of the Psyons that may or may not also be hit by the same publishing limitation when they try.

It would be incredible for me to not only release my timer skip Trials of the Succubi versions on Milovana, but all the other content that has been added like the upcoming 4.0!

[nocnoc]

I would like to second this. Milovana is such a unique and special website and community. Some creators such as Indyc and myself are really trying to bring the content offered to another level. Currently, the JSON file for The Legend of the Psyons (a huge RPG Estim game for Eos) is 20 megabytes. For some reason, even with a larger file size, I haven't had the same editing issues that Indyc has experienced, I still worry about publication issues later down the line. Right now, I just have a few hundred people testing the game via private link as the project is developed, but obviously I'll want to publish a 'public' alpha at some point.

[Trusfrated]

It would be incredible for me to not only release my timer skip Trials of the Succubi versions on Milovana, but all the other content that has been added like the upcoming 4.0!

Just adding my two cents that it would be great if the tease size limitation that Indy keeps running into could be lifted or effectively resolved. There's so much creativity and content out there that I'm sure many Milovana members like myself would love to see and experience!

Please consider this, Seraph0x!

[seraph0x]

I'd love to see you guys take webteases to the next level! There are some other things I'm working on at the moment that I have to finish first but maybe we can use the time to think about how this could work.

Supporting huge teases in the editor is actually the easy part. But I'd have to support them in the tease viewer as well. And some people have pretty old computers or slow internet so I have to think about how to make it work for them.

Here are some options:

Option 1: Tease Segments

I could make it such that you can split your tease into segments. There would be a special command to jump from one segment to another.

Pro: Teases overall could be infinitely large (though each segment would still be limited)
Pro: Easy to migrate existing teases - they'd just be teases that happen to only have one segment
Pro: Might be possible to share segments across teases - for example, imagine someone made an amazing estim calibration section and shared it so you can just jump to it.
Con: There may be a loading screen when moving from one segment to the next
Con: Authors would have to think about the structure and make sure individual segments aren't too big
Con: You could only use images that are in the current segment so you may have to upload the same images to multiple segments - I could see that getting annoying.

Option 2: Streaming Teases

Alternatively, I could change the viewer to load pages on demand. Right now, the page data for the whole tease is loaded upfront. Just like the viewer currently tries to guess upcoming pages and load the images, it could also preload pages.

Pro: Infinite teases with no subdivisions. It's like a big open-world game with no loading screens.
Con: Likely very tricky to implement - a lot of Eos assumes that the page data is always available
Con: Big changes like this are more likely to break existing teases
Con: Loading pages individually would cause a ton of requests to the server. I would likely have to group pages into chunks.

Option 3: Optimization

Instead of building a whole fancy engine change, I could also just tweak and optimize what we have now. For example, I could come up with an efficient binary format for teases which would allow a large tease to be orders of magnitude smaller than it is now.

Pro: Much easier to implement that either of the other options
Con: Wouldn't allow infinite teases, just significantly bigger ones
Con: Large teases would still have to be loaded upfront which may take a long time

[indyc]

I'd love to see you guys take webteases to the next level! There are some other things I'm working on at the moment that I have to finish first but maybe we can use the time to think about how this could work.

Oh wow wow! This is so exciting! I'm not sure I entirely understand the first 2 options but this is my uneducated opinion:

When you have the time, why not work on option 3. We can see how far that opens things up and then later option 1 or 2 could be added on if needed?

It sounds like the lowest hanging fruit in my mind and wouldn't it also improve the other two options should you need to go that far?

Thank you ever so much for maintaining everything much less improving it!

[nocnoc]

I'd love to see you guys take webteases to the next level! There are some other things I'm working on at the moment that I have to finish first but maybe we can use the time to think about how this could work.

Hi Seraph0x. It is great to hear that you think there may be a few options with regard to supporting larger Eos projects. It can be a little hard to evaluate the proposals without knowing the details of the behind the scenes mechanics, but I'm happy to share my thoughts. First it may be best to better explain the possible issues as they currently stand.

I think a few of Indyc's projects and my Legend of the Psyons (LOTP) RPG are some of the larger Eos endeavors that I am aware of. We have had some of the same issues, but also, he has experienced some problems that I have not. This could be because our projects are very large in different ways, and this may be worth noting when considering optimization options. I think Indyc's projects have more images and a greater total number of pages. Mine, on the other hand, features fewer pages that have a more complex structure that integrate with a huge JavaScript init "game engine". I have a lot of images, but not as many as Indyc. Before the last site update (I think almost a year ago?) we both experienced a lot of time out issues while uploading JSONs to an existing tease (such as when I would update the alpha tester tease). I started having issues with this when my JSON hit about 15 megs. Since the last site update, I've had fewer problems, even with a JSON at 20 megs, but I haven't tried to publish an alpha yet. Perhaps soon. My understanding is that publishing is far more prone to timing out and other sorts of errors for large projects. So I'm worried about that. On the user experience end, most of my testers report fairly smooth sailing. I have noted that when Milovana is hiccupping, a sound file may on a rare occasion fail to stop playing correctly. I've had redundant code in place for a while to help ensure that an estim pain signal doesn't get left on inadvertently. This has mostly mitigated the issue, but I do have it still reported once in a blue moon. I also make it clear that LOTP isn't really meant to be played on underpowered tablets or the like. Unfortunately Indyc has had some issues even after the last site update. Perhaps due to the different structure of his projects as described earlier. Indyc and I have exchanged JSONs and we can confirm that the issues are not hardware or Internet connection specific. I tried to get one of his old projects working to no avail and could not track down the problem.

As for the three options, I concur with Indyc that option 3 (optimization) sounds most promising and may be all that is needed. An OOM or two would go a long way. Also, anything that could help prevent timing out would be great. Option 1 (loading zones) also sounds like a very nice idea. One thing I would ask if that option is explored would be a shared resources section (so that some pictures and sound files could be accessed by any zone) and that the init would also be shared across zones. Otherwise, code maintenance would become unwieldy. The shared segments also sounds very interesting. For example, I have a unique copy-and-paste text-based (hex) save system that allows my game to have huge save files. Stuff like that may be nice to easily share with other creators. Option 2 would hopefully not be necessary, but if it were, perhaps it could be implemented as a whole new system like Eos-2 or something, so as not to break existing content.

I hope this is helpful. Thanks for an amazing site.

[ritewriter]

FWIW I'd also support optimization as the first thing to pursue. It seems like it'd be the least invasive for existing teases, and while it might be not infinitely large, it sounds like it should accommodate 99.9% of the teases out there (in other words, everything but TotS. )

[oneiromantica]

I don't have much insight into the player's architecture, but is it really the player that's the limiting point?

My experience with TotS is, that versions too big to work on still can be imported once and then the (pre-)viewer works perfectly fine. Problems only occur once we try to edit/publish the teases; this will result in backend errors once the editor tries to save the changes.

From what I can see, there's a simple reason to explain that behaviour: To bypass the current size limitations, we "compress" the JSONs by removing white space. This decreases file size by multiple MB, the files can be stored. If we export the tease again, we will get nice human readable versions which is as big as it was before the compression, so I deduct the JSON get beautified at some point during the persistence step, thus destroying all optimisations and becoming un-saveable again.

20MB is not that much of a file size nowadays, can be parsed reasonably quick (ok, I'm writing from a high-end dev machine here, but it works also really fast on e.g. my old phone), and will not require big amounts of system memory. I agree with low-bandwidth scenarios, but the asset size outweighs the JSON's by two orders of magnitude, if we take TotS as an example again, so the only issue would be longer loading times.
Still, even asset loading is not much of an issue, browser logs show that assets (at least images, didn't try to identify sound files) are only pre-fetched on a per-page basis, so that should not be a limiting factor at all and scale without any changes.

Of course, from the user's view it's not clear at all, why trying to save bigger files leads to server errors, so I have to guess from afar. Those remote observations give me a feeling that the eos-player could cope with at least reasonably bigger teases if they could just be transferred to the server.
Of course that could be wrong and the problems would begin to appear with even slightly bigger teases than those we can upload with workarounds...

I would second the suggestions that optimisation should be the preferred way; the others would require big changes/rewrites of the engine and introduce many complicated scenarios, especially the streaming one (of course, that one is the most intriguing :D ).