Introducing transformation modes: Snaplet's solution for simpler, safer data

Coding against realistic data is fantastic, but it can often feel like you're teetering on a tightrope when trying to get that data into your local development environment. One misstep can lead to catastrophe. That's where Snaplet comes in - we're here to be your safety net, making it a breeze to work with production-realistic data. And with our new transformation modes, we're taking simplicity and safety to new heights!

If you're new to Snaplet, here's a rapid rundown of how our system works:

1. We connect to your database (it could be staging, QA, or prod) with a connection string.
2. We introspect the database structure to understand your schema and data.
3. We let you subset your data, capturing only a portion or sample because, let's be real, you don't need to code against your entire database.
4. Now, prepare for some magic! We capture your data into a snapshot while transforming it. We change individual values but keep the overall "shape" intact, ensuring you get production-realistic data that's safe to use.
5. Finally, you can restore the snapshot (now a smaller, transformed sample) to your local development environment and code like a pro!

Previously, Snaplet users had to specify which columns in their tables needed transforming during the snapshot capture process. Although this offered a lot of control, it was time-consuming, and sensitive columns could be accidentally overlooked. To make life easier, we're introducing three new transformation modes:

1. Unsafe: Data for columns not specified in the config are copied as is, without being transformed.
2. Strict: Fails the capture if any columns, tables, or schemas not specified in the configuration are detected.
3. Auto: Automatically transforms data for columns, tables, or schemas not specified in the configuration.

Unsafe mode is how we used to do transformations, and still works as it did. It works best if you're very familiar with the data in your database and are comfortable that no potentially sensitive data can slip through.

Strict and Auto are the hot new modes on the block. Strict mode is perfect for those who want total control. If any unspecified data is detected, the capture will fail, keeping your data extra secure. This is the safest way to use Snaplet transformations because you have complete control over every transformation, and nothing happens without you defining it. When a capture fails, you’ll be alerted, and can decide what to do next.

Auto mode strikes a fantastic balance between Unsafe and Strict, executing specified transformations while automatically and safely transforming any unspecified columns. This gives you complete peace of mind that you're getting production-realistic data that's entirely safe to use.

But wait, there's more! Auto mode's transformations are so smart, your transformed data will still resemble the original. How? Snaplet analyzes column types and 'shape' to ensure transformed data makes sense within the context of your database. For example, if the column in question is an enum type, the resulting transformations will still be valid enum values for that enum type. Similarly, we also examine the 'shape' of the column: a column may be of type text in the database, but if the name of the column is email, Snaplet will infer that the column represents email addresses and account for this when transforming.

‍

// .snaplet/transform.ts
import { copycat } from '@snaplet/copycat';
import type { Transform } from './structure';

export const config: Transform = () => ({
  $options: {
    mode: 'auto',
  },
});

If you’re using our onboarding in Snaplet Cloud to capture your first snapshot, you’ll be presented with the option to select whether your data are transformed as part of the snapshot creation process:

Opting to skip transformation will, skip the transformations, while selecting transform will set your transform mode to auto.

snaplet snapshot capture --transform-mode=auto

# or the shorthand form:

snaplet snapshot capture -t auto

For more information, check out our docs. It’s worth noting that there are a few limitations around auto mode - more on this here. Also, we're still working on optimizing auto mode for colossal datasets, so bear with us!

We'd love to hear your thoughts on our new transformation modes, and if you need support for specific use cases, don't hesitate to reach out on Discord.

So, step up to the high-wire stage and let the transformations commence! Embrace the power of Snaplet's spectacular transformation modes and shape your data safely and swiftly. With these new tools at your disposal, you'll be walking the tightrope of coding against production-realistic data like a seasoned acrobat in no time! 🎪

Happy transforming, folks! 😸

‍