EOS TeaseStorage limit

[Philocalist]

Hey there,
I've had a similar thread to find out the limit of the eos storage limit, or how to solve it.
The problem is, that there seems to be an overall limit of around a 100 characters. The variable names take up space too, but I need much more. Even with compression those 100 characters are not enough for me. Could anyone help who has had a lot of variables to save on their tease? Also, if there would be an option in eos to view and delete teaseStorage items it would be great.
I am panicking right now, since the tease I'm making probably will end up being the largest ever and if I can not utilize the saving function I have to make the user save a large string before they leave and to paste it in when they come back. And use some hashfunction (that I might store) to check the validity of the string.
Any thoughts?

[undeniable_denial]

I've just looked into this myself and here is what I found:

1. The limit is 1024 characters

The limit is NOT 1024 bytes (or 1 kibibyte). For example the character "⅝" is made up of 3 bytes in UTF-8 and I was able to store just as many of them as single-byte characters. This could potentially be abused for "compression".


The limit applies to all storage items combined. There is also overhead for quotation-marks, assignment operators, commas, brackets, etc. which also take up space. Here are two examples that fill up the entire available space:


Example 1: storage0 contains 1009 x "⅝". Together with the characters {"":""} (7 characters) plus the variable name (8 characters) the sum is 1024 = 1009 + 7 + 8

Code: Select all

{"storage0":"⅝...⅝"}

Example 2: storage0 contains 992 x "⅝" and storage1 contains "bbb". A lot more is lost to overhead:

Code: Select all

{"storage0":"⅝...⅝","storage1":"bbb"}

2. There are functions not mentioned in the API-docs

Code: Select all

length			-- returns the number of stored items
​key ( <number> )	-- returns the key (name) of the n-th stored item
​removeItem( <string> )	-- probably removes item from storage (haven't tested)
​clear()			-- probably clears the entire storage (haven't tested)​

3. How to deal with this?
Well, it depends what you are trying to do. Here are a couple of ideas:

• Minimize the length of variable names and storage keys
• Minimize the number of stored items to save on overhead
• Don't store anything you can derive from something that is already stored.
• Choose an efficient way to store your data. Possibly bit fields.
• You could use simple compression algorithms, such as huffman coding to try and compress your data to fit into 1024-7 = 1017 bytes. Probably overkill in most cases.

Regarding bit-fields. I recall that you're working on a kind of adventure, but I don't know any specifics. Let's assume you have several items that can be acquired then you could store which items the player has in a bitfield:

Code: Select all

00110010	-- 1 byte storing the information that player has items with IDs 2,5,6
87654321

That's just an idea.


Honestly, your strategy to finalize your variables first and then figure out how to store them is a good one.

[Philocalist]

.

[Philocalist]

.

[Philocalist]

I've just looked into this myself and here is what I found:

1. The limit is 1024 characters

The limit is NOT 1024 bytes (or 1 kibibyte). For example the character "⅝" is made up of 3 bytes in UTF-8 and I was able to store just as many of them as single-byte characters. This could potentially be abused for "compression".


The limit applies to all storage items combined. There is also overhead for quotation-marks, assignment operators, commas, brackets, etc. which also take up space. Here are two examples that fill up the entire available space:


Example 1: storage0 contains 1009 x "⅝". Together with the characters {"":""} (7 characters) plus the variable name (8 characters) the sum is 1024 = 1009 + 7 + 8

Code: Select all

{"storage0":"⅝...⅝"}

Example 2: storage0 contains 992 x "⅝" and storage1 contains "bbb". A lot more is lost to overhead:

Code: Select all

{"storage0":"⅝...⅝","storage1":"bbb"}

2. There are functions not mentioned in the API-docs

Code: Select all

length			-- returns the number of stored items
​key ( <number> )	-- returns the key (name) of the n-th stored item
​removeItem( <string> )	-- probably removes item from storage (haven't tested)
​clear()			-- probably clears the entire storage (haven't tested)​

3. How to deal with this?
Well, it depends what you are trying to do. Here are a couple of ideas:

• Minimize the length of variable names and storage keys
• Minimize the number of stored items to save on overhead
• Don't store anything you can derive from something that is already stored.
• Choose an efficient way to store your data. Possibly bit fields.
• You could use simple compression algorithms, such as huffman coding to compress you data into 1024-7 = 1017 multi-byte characters. Probably a bad idea in most cases.

Regarding bit-fields. I recall that you're working on a kind of adventure, but I don't know any specifics. Let's assume you have several items that can be acquired then you could store which items the player has in a bitfield:

Code: Select all

00110010	-- 1 byte storing the information that player has items with IDs 2,5,6
87654321

That's just an idea.


Honestly, your strategy to finalize your variables first and then figure out how to store them is a good one.

This is extremely helpful, thanks!
I really should finalize everything before writing an algorithm to save them. But it seemed like such a risk at first. Now I have an inelegant plan b. And I'm pretty sure that even with the best compression I can't make all the variables to fit into a 1024 chars. Although, if we are talking 1024 utf-8 characters, and I only use acsii then with some next level wizardry I might. Is it worth looking into that even?
Also is there a way I can empty my teaseStorage in the editor? Seems like I made some junk while testing...

[undeniable_denial]

• You could use simple compression algorithms, such as huffman coding to compress you data into 1024-7 = 1017 multi-byte characters. Probably a bad idea in most cases.

And I'm pretty sure that even with the best compression I can't make all the variables to fit into a 1024 chars. Although, if we are talking 1024 utf-8 characters, and I only use acsii then with some next level wizardry I might. Is it worth looking into that even?

I have worded this badly. In hindsight I'd say don't bother with multi-byte characters. I'm not sure how useful this utf-8 trickery really is and it's probably not worth the research effort.
Compression is worth it if you're half way there, but can't shave off anything else. JSON-data is essentially text and text has crazy compression rates, however, this is still only moving the limit, not removing it.
I don't think you will need compression, though. A thousand characters can hold a lot of information.

Also is there a way I can empty my teaseStorage in the editor? Seems like I made some junk while testing...

As I wrote in paragraph 2, there are some undocumented functions, one of which is teaseStorage.clear() . I believe it should do the trick.

[Philocalist]

• You could use simple compression algorithms, such as huffman coding to compress you data into 1024-7 = 1017 multi-byte characters. Probably a bad idea in most cases.

And I'm pretty sure that even with the best compression I can't make all the variables to fit into a 1024 chars. Although, if we are talking 1024 utf-8 characters, and I only use acsii then with some next level wizardry I might. Is it worth looking into that even?

I have worded this badly. In hindsight I'd say don't bother with multi-byte characters. I'm not sure how useful this utf-8 trickery really is and it's probably not worth the research effort.
Compression is worth it if you're half way there, but can't shave off anything else. JSON-data is essentially text and text has crazy compression rates, however, this is still only moving the limit, not removing it.
I don't think you will need compression, though. A thousand characters can hold a lot of information.

Also is there a way I can empty my teaseStorage in the editor? Seems like I made some junk while testing...

As I wrote in paragraph 2, there are some undocumented functions, one of which is teaseStorage.clear() . I believe it should do the trick.

I checked, the teaseStorage.clear() does work, and is liberating. I looked at the source code and couldn't find the parts where the undocumented functions are. And the sad thing is that I have more than 1500 variables, so storing that in 1024 chars will be a challenge. Taking your advice, I will postpone this until my vars are finalized.
Thank you again for your help!

[Jackall]

OooohKay then its time for me to rename everything into shorts and find where everything related to everything is, one by one...

edit: done it in a few hours and
teaseStorage.clear() works (despite red underline). thanks!