ETS - EOS Text Storage - Solution for large save data

[alexfayer]

Hello tease creators,

I have come up with a solution to go around the storage limitations of EOS. The idea is to use a text or word file to store the data. So the user has to save the data himself on his computer.

The data is not easily editable so it is not easy for the average user to manipulate the savegame, but of course it is far from impossible. There are improvements possible to prevent more experienced user from cheating like adding a CRC check or a hash, but if you can manipulate the current system the chances are very high that you could also do that if there is a hash present. I created a small test so you can try it and tell me any bugfixes or improvements I should make.
ETS - EOS Text Storage Test

I could create multiple megabytes of data with it. But depending on your hardware it will take some time to export a lot of data. 10.000 values works fine without a noticeable delay for me. 100.000 will take some time though.

I published the code below if you want to use it in your own tease. Just paste it into your init script to use it.
I also added the EOS test as attachment so you can import this into EOS and see with examples on how to use it.

Documentation

etsSaveValue(key, value)
Saves a key with a value. The value can also be an object.
Example: etsSaveValue("name", "alex")

etsLoadValue(key, defaultValue)
Loads the value of a given key and returns it. If the key does not exists the key is created with the default value and the default value is returned.
Example 1: etsLoadValue("teaseDurationSec", 120);
Example 2: etsLoadValue("name", null);

etsExport()
Returns the string to be saved by the user.

etsImport(importString)
Imports the previously exported string and initializes the etsJsonObject.
Warning: This will delete the current saved data!
Returns null on success. On error a message of the error is returned.
Example: etsImport("##01D6D0==")


ETS version 1 - 24.05.2020 :

Spoiler: show

Code: Select all

// === ETS - EOS Text Storage === BEGIN
var etsJsonObject = JSON.constructor();
var etsDebug = false;  // change to true for debugging
var etsPrefix = "##";
var etsSuffix = "==";  // should be different from prefix for better error checking
var etsXorValue = 0xAD; // change to something else if you want. Changing this corrupts current existing savedata from other users.
var etsVersion = 1;

/**
 * @brief Saves the value of the given key
 */
function etsSaveValue(key, value)
{
  etsJsonObject[key] = value;
  
  if(etsDebug)
    console.log("ETS: Saved " + key + " = " + value);
}

/**
 * @brief Loads the value of the given key and returns it.
 * If the loaded value is undefined the value of the given key will be set to the given default value.
 * 
 * @return  the value of the key or defaultValue if no set value exists.
 */
function etsLoadValue(key, defaultValue)
{
  var etsValue = etsJsonObject[key];
  
  if(etsValue === undefined)
    etsValue = defaultValue;
  
  etsJsonObject[key] = etsValue;
  
  if(etsDebug)
    console.log("ETS: Loaded " + key + " = " + etsJsonObject[key]);
  
  return etsJsonObject[key];
}

/**
 * @brief Exports the save data as a string
 * 
 * @return  The exported string
 */
function etsExport()
{
  var etsJsonString = JSON.stringify(etsJsonObject);
  
  var etsByte;
  var etsByteString;
  var etsHexString = etsPrefix;
  
  // add version byte
  etsByteString = etsVersion.toString(16).toUpperCase();
  if(etsByteString.length < 2)
    etsByteString = "0".concat(etsByteString);
    
  etsHexString = etsHexString.concat(etsByteString);
  
  // json -> byte -> XOR -> hexString
  for(var i = 0; i < etsJsonString.length; i++)
  {
    etsByte = etsJsonString[i].charCodeAt();
    etsByte ^= etsXorValue;
    etsByteString = etsByte.toString(16).toUpperCase();
    
    if(etsByteString.length < 2)
      etsByteString = "0".concat(etsByteString);
    
    etsHexString = etsHexString.concat(etsByteString);
  }
  
  // add suffix
  etsHexString = etsHexString.concat(etsSuffix);
  
  if(etsDebug)
    console.log("ETS: Exported \n[" + etsHexString + "]");
  
  return etsHexString;
}

/**
 * @brief Imports the save data from the given string
 * 
 * @return  null on success, or a string describing the error
 */
function etsImport(importString)
{
  // remove whitespaces
  importString = importString.split(" ").join("");
  importString = importString.split("\t").join(""); // tabs
  importString = importString.split("\n").join(""); // new line - should be filtered by eos
  importString = importString.split("\r").join(""); // carriage return - should be filtered by eos
  
  // check string length
  var etsMinLength = 6 + etsPrefix.length + etsSuffix.length; // 6 = version(2) + '{' + '}'
  if(importString.length < etsMinLength)
    return "Text storage must contain at least " + 
    etsMinLength + " characters. Received: " + importString.length +".";
  
  // Check for prefix and suffix
  var etsIndexBegin = importString.indexOf(etsPrefix);
  var etsIndexEnd = importString.lastIndexOf(etsSuffix);
  if(etsIndexBegin == -1)
    return "Missing prefix \"" + etsPrefix + "\".";
  if(etsIndexEnd == -1)
    return "Missing suffix \"" + etsSuffix + "\".";
  
  // remove prefix and suffix from string
  var etsHexString = importString.substring(etsIndexBegin+etsPrefix.length, etsIndexEnd);
  
  // check for valid hex string
  if(etsHexString.length % 2 !== 0)
    return "Text storage must contain an even amount of characters.";
  if(etsHexString.length < 6)
    return "Text inside prefix and suffix must be at least 6 characters long.";
  
  
  // check for matching version - later adding backwards compatibility here
  var etsParsedVersion = parseInt(etsHexString.substring(0, 2), 16);
  if(etsParsedVersion != etsVersion)
    return "Version " + etsParsedVersion + " not supported. Active ETS version: " + etsVersion;
  
  // convert hex string to a json string
  var etsByte;
  var etsJsonString = "";
  
  for(var i = 2; i < etsHexString.length; i+=2)
  {
    var etsSingleByteString = etsHexString.substring(i, i+2);
    etsByte = parseInt(etsSingleByteString, 16);
    etsByte ^= etsXorValue;
    etsJsonString = etsJsonString.concat(String.fromCharCode(etsByte));
  }
  
  // parse string into json object
  try
  {
    etsJsonObject = JSON.parse(etsJsonString);
  }
  catch(err)
  {
    if(etsDebug)
    {
      console.log("ETS: Invalid JSON \n[" + etsJsonString + "]");
      console.log(err.toString())
    }
    
    return "Invalid data";
  }
  
  if(etsDebug)
    console.log("ETS: Imported \n[" +  JSON.stringify(etsJsonObject) + "]");
    
  return null;  // success
  
}
// === ETS - EOS Text Storage === END

[RemiHiyama]

There are improvements possible to prevent more experienced user from cheating like adding a CRC check or a hash, but if you can manipulate the current system the chances are very high that you could also do that if there is a hash present.

My advice would be to not bother with further anti-cheat mechanisms. It's almost impossible to make them work in a fully open-source client-side environment like EOS is, and as essentially single-player games it's a lot of work for very little gain.

Instead, if people are enjoying your content in ways you did not intend, I suggest taking the win.

[fapnip]

My advice would be to not bother with further anti-cheat mechanisms. It's almost impossible to make them work in a fully open-source client-side environment like EOS is, and as essentially single-player games it's a lot of work for very little gain.

Instead, if people are enjoying your content in ways you did not intend, I suggest taking the win.

Cheating wouldn't be my main concern with user sourced save sate. Putting the tease in an unintended state via corrupt restore would likely lead to all sorts of invalid bug reports and poor user experience. Would be much better to respond with "Sorry, that save state is buggered. Try again."

[nocnoc]

So I did a quick test adding this to Legend of the Psyons and I can confirm it works like a charm. I only saved and loaded the player object for now, which is only a few pages of text data. This is a complex object though, containing other objects like various weapons and the like. Later this week, I'll tinker with getting the world data stored, which will be much more substantial, but if it works for the player object, it will work for the world data objects. I may have to reorganize a few things, but right now my code is still under 7,000 lines so it won't be too bad. So as long as people are going to be willing to copy a text output, save it in Notepad or Word or some other text program and paste it back in, then there will be a save option of some sort for my huge game project and I can go back to the original design intention of a 10 to 30 hour playthrough. I'll update later this week with how the world data goes. (This data increases in size as the player travels as much of the data is generated on first visit to a location, whether it is a normal location or a procedurally generated location.)

Thanks to Alex for the good work!

[alexfayer]

That's great to hear! Using complex object means you only need to call loadValue() once, which is very handy.

This should open up the possibility of creating much longer and more complex teases. Also saving the game, reloading the tease and importing the save again should free up memory. This is a temporary "fix" for the current EOS memory leak.