Improved backend logging

On-premise customers have an easier time monitoring Retool apps with improved backend logging. Each log emission is now a JSON object with tracing information to track what happened across an entire request. We also include useful information like the response time for each query and whether or not it succeeded.

This logging improvement will help you connect a monitoring service (i.e. Datadog) to read the logs and build reports and alerts around them.

Here’s an example of a success and failure case:

// SUCCESS

{"requestId":"8aa9df50-78bc-4f95-b1f5-eb798c57888b","email":"yogi@retool.in","organizationId":1,"organizationName":"yogi@retool.in","message":{"type":"REQUEST","method":"POST","url":"/api/pages/uuids/3e9e40fa-ad86-11e9-a2e7-bf5202eb3fb8/query?queryName=query2","hostname":"http://localhost:3000"},"level":"info","timestamp":"2019-07-29T17:47:41.646Z"}
{"requestId":"8aa9df50-78bc-4f95-b1f5-eb798c57888b","message":{"type":"QUERY_REQUEST","ipAddress":"::ffff:172.18.0.1","userSid":"user_4c424358408e43a2bd4757e4dfce83c9","resourceName":"hammerhead_production","resourceType":"postgresql","environment":"production","isPreview":false},"level":"info","timestamp":"2019-07-29T17:47:41.741Z"}
{"requestId":"8aa9df50-78bc-4f95-b1f5-eb798c57888b","email":"yogi@retool.in","organizationId":1,"organizationName":"yogi@retool.in","message":{"type":"QUERY_RESULT","success":true,"isPreview":false,"pageUuid":"3e9e40fa-ad86-11e9-a2e7-bf5202eb3fb8","queryName":"query2","resourceName":"hammerhead_production","resourceType":"postgresql","environment":"production","queryResponseTimeMs":"27.0771"},"level":"info","timestamp":"2019-07-29T17:47:41.771Z"}

// FAILURE

{"requestId":"91497c47-22d2-4417-8b8b-ad5bc2054231","email":"yogi@retool.in","organizationId":1,"organizationName":"yogi@retool.in","message":{"type":"REQUEST","method":"POST","url":"/api/pages/uuids/3e9e40fa-ad86-11e9-a2e7-bf5202eb3fb8/query?queryName=query1","hostname":"http://localhost:3000"},"level":"info","timestamp":"2019-07-29T17:47:49.426Z"}
{"requestId":"91497c47-22d2-4417-8b8b-ad5bc2054231","message":{"type":"QUERY_REQUEST","ipAddress":"::ffff:172.18.0.1","userSid":"user_4c424358408e43a2bd4757e4dfce83c9","resourceName":"hammerhead_production","resourceType":"postgresql","environment":"production","isPreview":false},"level":"info","timestamp":"2019-07-29T17:47:49.482Z"}
error: duplicate key value violates unique constraint "blocks_name_organizationId_key"
    at Connection.parseE (/retool_backend/node_modules/pg/lib/connection.js:601:11)
    at Connection.parseMessage (/retool_backend/node_modules/pg/lib/connection.js:398:19)
    at Socket.<anonymous> (/retool_backend/node_modules/pg/lib/connection.js:120:22)
    at Socket.emit (events.js:189:13)
    at Socket.EventEmitter.emit (domain.js:441:20)
    at addChunk (_stream_readable.js:284:12)
    at readableAddChunk (_stream_readable.js:265:11)
    at Socket.Readable.push (_stream_readable.js:220:10)
    at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)
{"requestId":"91497c47-22d2-4417-8b8b-ad5bc2054231","email":"yogi@retool.in","organizationId":1,"organizationName":"yogi@retool.in","message":{"type":"QUERY_RESULT","success":false,"isPreview":false,"pageUuid":"3e9e40fa-ad86-11e9-a2e7-bf5202eb3fb8","queryName":"query1","resourceName":"hammerhead_production","resourceType":"postgresql","environment":"production","queryResponseTimeMs":null,"errorMessage":"duplicate key value violates unique constraint \"blocks_name_organizationId_key\""},"level":"info","timestamp":"2019-07-29T17:47:49.491Z"}
Error in a microservice { error: true,
  message:
   'duplicate key value violates unique constraint "blocks_name_organizationId_key"' }

Support for pushing groups from Okta, Ping Identity, OneLogin, etc

You can now manage groups in the same place as you do for everywhere else and have the changes sync to Retool:

• Assign new users to groups in Retool when you provision their accounts in Okta.
• Remove users from groups in Retool when you do so in Okta.
• Create new groups in Okta that sync with Retool.
• Destroy group in Okta that sync with Retool.

Improved navigation between different elements in Edit Mode

• Retool now highlights the selected element (i.e. query, component, temporary state, or transformer) in the left-hand panel model browser.
• Also, each time an element is referenced in a query or text input, a hyperlink appears under it. This hyperlink will reveal the elements details and is a quick shortcut to edit it.

Expose column indexes for faster querying

You’ll now see a ⚡ icon next to column names in the query panel database schema when that column has an index. Queries that use indexed columns often run faster than they otherwise would.

Descriptions for releases

Now you can add a description when you create a new release (aka version) of an app. That way if you have to roll back to an old version, you’ll know which version to pick.

Improved SQL autocomplete

The query editor autocomplete now handles handles camelcase, non-lowercase letters, numbers, and underscores in table names.

New & Improved

• Added an optional MaxKeys parameter to AWS S3 queries to limit the number of objects returned
• Updated Google Sheets integration to read, append, and update beyond 26 columns
• Updated Datetime Picker to re-trigger queries after selecting a new date or time
• Updated sign in button to read “Sign In” for customers with SSO enabled
• Added icons to query tabs to give you a quick visual scan for which resource they use
• Fixed the Bounding Box component to calculate the correct (x,y) coordinates for landmarks.
• Fixed JSON Schema forms to respect when form elements have a required flag
• Form submit buttons now prevent accidental multiple submissions
• Improved performance of Javascript queries

Transformers for query data

Queries now have built-in transformers, which let you write JS to “transform” the result of any query. For example, if your API returns data in a column-based format, but you want to “transform” the data into an array of objects, you can write a transformer directly in the query. Then, when you use {{ query1.data }} elsewhere, it’ll be transformed, so you won’t have to write the same JS in multiple places.

• Click “enable” next to the “Transformer” section below your query.
• Use data in the Query Transformer to access the query results.
• In the rest of the app, access the transformed results with the data property inside {{ }}, for example with {{ query1.data }}.
• If you need to, you can still access the original response of the query with {{ queryName.rawData }}.

Note: you can disable the query transformer by clicking the “disable” button.

Example of using the Transformer to filter User IDs with a dynamic slider.

Click to copy the green “evaluation preview”

Retool now supports copying the content from an “evaluation preview” textbox. ​​

​​For example, in the GIF below, we’re copying the actual live results of a query into a JSON Explorer, so we can inspect it.

Copying the green “evaluation preview” text to inspect the data in a JSON explorer component.

Copy & paste containers with their children

You can now copy and paste Containers with of their children.

• Select the parent Container (or Tabbed Container) component.
• Copy with ⌘ + C on Mac (or CTRL + C on Windows / Linux).
• Click on the canvas to deselect the Container.
• Paste with ⌘ + V on Mac (or CTRL + V on Windows / Linux).

Copy and paste containers with their children!

Paste components inside existing containers

Also, you can also now paste components directly into Container or Tabbed Container components.

• Click the component and copy it with ⌘ + C on Mac (or CTRL + C on Windows / Linux).
• Select the Container or individual tab of a Tabbed Container where you want the copy to go.
• Paste the component with ⌘ + V on Mac (or CTRL + V on Windows / Linux).

Copy and pasting a button from one container to another.

New & improved

Performance enhancements

Last week, we spent a lot of time working on making Retool faster; now some of that work has been released. Edit mode and Presentation mode are faster, more complex apps should feel snappier, and even tables with custom cell colors & mapped values will load quicker too!

New minor features

• Added ability to control staging vs. production environment with a URL parameter.
• Added an onboarding checklist for new users! (If you complete it, we’ll mail you a Retool T-shirt 👕).
• Updated visual cue to make it more obvious when components reference unsaved queries.
• Added sms:// and facetime:// to our list of allowed protocols in HTML links.
• Added option to disable introspection for GraphQL resources.
• Added ability to specify an audience parameter in OAuth 2.0 authentication for HTTP and GraphQL APIs.
• Added ability to clear the PDF Viewer component programmatically with .clear().
• Added better support for newlines when copying text and components to the clipboard.
• Added option to hide the clear button in Select components (aka Dropdown components).
• Added option for scroll on the Modal component.

Small fixes

• Fixed the the display for the “Always show download button” option on Table components.
• Fixed upsert action for querying Snowflake databases.

Copy and paste components between apps

• Copy & paste entire components from one app to another!

Visual cue for staging environment

• Added a yellow banner for both Editor and Presentation modes when staging mode is selected
• Only available for apps that have a staging environment configured

New

• Start, stop, resume, and reset a stopwatch-style timer

New & Improved

• Added triggeredby``Id attribute to manually-triggered queries so you know what triggered it
• Added button to Settings > Advanced page to download Retool’s public key for git syncing
• Added support for auto-pagination for DynamoDB resources
• Added ability to specify GCS bucket in resource editor
• Added ability to specify more than 1 permitted file type for S3 Uploader components
• Added onchange query trigger for Radio Group components
• Added write-back capability for BiqQuery resources
• Added setValue function to Editable Text component
• Added autocomplete for JS query utility libraries
• Now displaying hints for common SQL query gotchas in query editor
• Various performance improvements for big Retool apps
• Better UX in Query Playground for long queries

Concurrent developer warnings

• Added a notification to the header that shows who else is modifying the same Retool app.
• You can use this to avoid overwriting changes that your team mates are making to the Retool app.

Deleting apps

• Added ability for admins to permanently delete apps that are in the archive folder.

New

• Added the option to specify the ACL for files uploaded to Google Cloud Storage
• Added option to allow users to define custom values in Multiselects. Enable this by using the "Allow user defined values" option in Multiselects.

Resource access levels

• Extended the access control levels in Retool to allow admins to restrict access to Resources connected in Retool.
• You can use this feature to do things like:
  • Only allow analysts to use readonly database connections.
  • Only allow developers to use apis / databases that they manage.

Here, the group lets the "Awesome Intern" use the onboarding_db (readonly) and onboarding_api resources to create / modify apps

Query playground

• Added the Query Playground which lets developers write queries quickly without having to create an app.

• Coming soon: reusing queries you write in the playground within your Retool apps.

Tile homepage view

• Added a new tile-based home screen for Retool apps
• You can customize the image you use for each tile.
• You can also "favorite" your most frequently used apps so that always appear at the top of the page.

New

• Added support for deleting S3 files.
• Added support for S3 bucket URLs in APIs that use the AWSv4 authentication method.
• Increased maximum timeout for queries to 10 minutes.

Fixed

• Fixed how custom redirect flows in "Custom Auth" APIs work in "staging" environments.
• Fixed POST/PUT/DELETE requests in the Stripe integration that have an empty body.

Query Caching

• Added query caching as a way to optimize the performance of Retool apps.
• Queries can have its results cached for up to one day (the cache TTL)
  • The first time you run a cacheable query, it will run the query normally and then remember the query result.
  • Anytime anybody runs the query again the remembered result gets returned instantly.
  • The remembered result is deleted after the cache TTL expires.
• You can configure this option in the "Advanced options" section of the query editor.

Retool community!

• Visit https://tryretool.com/community to join the community 🙂

New and Improved

• Added lastUploadedFileName to the S3 Uploader component. Works similarly to the lastUploadedFileUrl.
• Added live warnings to make it easier to write Javascript scripts.
• Added a utils.downloadFile function to Javascript scripts. Use this to with API queries that return pdfs / docx / xlsx / etc. to download the files returned. Docs here.

Fixed

• Changed the MongoDB insertOne queries to also return the created object, including the MongoDB _id.
• Fixed deadlink on the Permissions dashboard.

RethinkDB Integration

• Added RethinkDB as a Retool integration.
• Run get, insert, update, delete, and find queries on your database.
• Let us know what else we can add.

Lambda Integration

• Added AWS Lambda as a Retool integration.
• Invoke lambda functions with whatever payload you need.
• See more info in our docs here.

Improved

• Added support for sharded Microsoft SQL Server databases.
• Improved the user management screen in Retool. Disabled users are greyed out, and we show accurate last active dates for each user.

Fixed

• Fixed how Custom Auth API integrations work in staging environments
• Fixed how non-string values are passed in FormData API requests.
• Fixed error that Slack webhooks would return in on premise Retool.
• Disabled overly aggressive cache headers in on premise Retool.
• Fixed editing JSON columns in editable tables.

Drag and drop

• We've enabled zooming to the drag and drop interface to make it easier to make big changes to your app.
• We've made it easier to re-arrange components on crowded applications.
• We've made it easier to drop components into existing containers.
• We've added resize handlers to all four corners of every component.
• We've made it easier to select multiple components.
• Here's a preview of some of these new features in a gif!

Better navigation

• We've changed our sorting algorithm for the Quick Switcher (use Cmd+K or Ctrl+K to bring it up!) to rank more relevant applications.
• We've added an interface to switch between apps to the End User mode.

Fun confetti!

• We've added a fun easter egg: make your queries fire off some confetti when it successfully runs!

Component changelog

Date Pickers and Date Range Pickers

• We've added restricting dates your users can select.
• You can define a minimum and a maximum date for both date pickers and date range pickers.

PDF Viewer

• We've added the PDF Viewer component. Use this to render PDFs for your users.
• You can either use a URL or a PDF file that's been encoded as base64.
• You can get a base64 encoded version of a PDF by using the Retool's REST Query.
• Here's an end-to-end example of how you might render a PDF from the IRS.

Checkbox Tree

• We've added a checkbox tree that makes it easy to select categories and subcategories.

Statistics

• We've added a statistic component to make it easy to render numbers like KPIs, etc.

Integrations changelog

Cassandra improvements

• We've added writing back to your Cassandra database.
• We've added support to connect to Cassandra using custom SSL CA Certificates.
• This feature works in the same way as other SQL Databases like Postgres, MySQL, etc.
• You'll need to create a new resource with a Cassandra user that has edit permissions.
• You can learn more about how that works here.

Snowflake improvements

• We've also added writing back to your Snowflake database.
• This feature works in the same way as other SQL Databases like Postgres, MySQL, etc.
• You'll need to create a new resource with a Snowflake user that has edit permissions.
• You can learn more about how that works here.

REST API Authentication

• We've added a "Form" step to the Custom Auth method.
• Use this to ask for credentials like usernames and passwords to pass to your API.
• Learn more by reading the docs here!

Uncategorized features

Fixed

• RadioGroup components now overflow correctly.
• Tables and other components in ListViews now render correctly.
• JSON Schema Form components with input:range (sliders) now work correctly.
• JSON Schema Form components that use the required now validate it's inputs correctly.
• Retool now correctly release and re-use database connections.
• GraphQL Queries now support Auth0 as an authentication method.
• MongoDB distinct queries now correctly validate inputs.
• BigQuery queries now format dates and timestamps correctly.
• New components created through copy paste now appear on the row below.

Improved

• We've improved the syntax highlighting for Custom Component editor.
• We've it easier to switch between mobile and desktop layouts in the drag and drop interface.
• We've released starter templates that make it easier to build apps in Retool.
• We've improved how Google Sheet queries.
• We've added support for connecting to Redis via SSL.
• We've added support for dynamic MongDB collection names.
• We've added a currency type to TextInput components.

Multiple Action Buttons

• Tables now can have multiple action buttons!

Cassandra Integration

• Retool now supports Cassandra v3+!
• We support using CQL - a SQL-like query language for Cassandra.
• See our docs here: https://docs.tryretool.com/docs/cassandra-integration

Google Cloud Storage Integration

• Retool now supports all the same configurations in S3 with Google Cloud Storage
• See our docs here: https://docs.tryretool.com/docs/google-cloud-storage-integration

Vertica Integration

• Retool also supports Vertica as a database!
• Reading and Updating are both supported.
• See our docs here: https://docs.tryretool.com/docs/vertica-integration

Map Improvements

• Added an onViewPortChange hook to trigger queries
• Added an onPointSelected hook to trigger queries
• Added .selectPoint(...) as as a method on Map components
• Improved accuracy of the custom marker positioning
• Made the Map component update its latitude and longitude property when the viewport updates.
• Added GeoJSON styling. Try this for an example!

{ type: 'line', paint: { 'line-width': 5 } } 

Signature Component

• Retool now has a Signature Component!
• It exposes a .signatureImageFile that you can use like the Filepicker's .file property to upload the image to services like S3 or even your own API.

Stripe Checkout Component

• Retool now has a component for collecting credit card information securely through Stripe's SDK.
• The component exposes a .stripeToken property that you can use with Stripe's API to create a credit card.

Date Range component

• Retool also has a date range component for selecting begin / end dates.

Various features and improvements

• Uploaded Excel files are also now automatically parsed!
• ListViews now have improved performance!
• Added support for deleting permission groups
• Added support for copying invite links directly from the Retool settings interface.
• Show staging / production toggle in the end user mode
• Support user-defined expiration timeouts for OAuth resources.
• Added utils.getCurrentPosition which returns a Promise that resolves to the device's current position
• Fixed bulk operations on Salesforce
• Fixed intermittent problems with CSV parsing on Windows machines
• Added option to render dates in Tables in their original timezone.

Custom React Components

• You can now create your own custom components using React and Javascript.
• We give you full access to the same APIs that our native components have.
• For example, you can now use whatever charting library you want (Plot.ly / D3 / etc.) as well as use custom React libraries like Material UI.
• See below for an example of a Plot.ly scatter chart in Retool. Full docs here

Dynamically Hiding / Showing Components

• Retool now supports dynamically hiding and showing components.
• Retool automatically relayouts the app to get rid of the empty whitespace.
• Check it out in action below! Full docs here

Undos / Redos

• Retool now supports undos and redos!
• On Macs use Cmd+Z to Undo and Cmd+Shift+Z to Redo changes you make while building your apps
• On Windows / Linux use Ctrl+Z to Undo and Ctrl+Shift+Z to Redo changes you make while building your apps
• Here it is in action:

Improved Calendar

• Added the option to modify and delete events in the calendar
• Added the ability to set what kind of interactions in the calendar are allowed (e.g. creating events, modifying events, deleting events, etc.)
• Added the ability to choose the default view of the calendar
• Added a preview for events selected

OAuth 2.0 Bugfixes

• Retool now support sharing OAuth 2.0 credentials between users.
• Retool also supports specifying an OAuth 2.0 Audience.
• We've greatly improved the reliability of our OAuth 2.0 integration and extended support to GraphQL as well.

On Premise Response Time and Error Monitoring

• Retool now ships with support for monitoring software.
• New Relic example config

NEW_RELIC_LICENSE_KEY=your-new-relic-license-key
NEW_RELIC_APP_NAME=retool-yourcompany-name
NEW_RELIC_NO_CONFIG_FILE=true

• Sentry example config:

SENTRY_DSN=your-sentry-dsn

Google Sheets

• We've added support for updating Google sheets.
• More docs and functionality coming soon!

Misc Features / Bugfixes

• Added support for triggering queries after a date is selected in the DateTimePicker with the onSelect property
• Added support for Delimiter in S3's List Objects operation
• Added support for GraphQL Cursor Based Pagination. Docs here!
• Added support for ordering in Firebase queries
• Added support for deleteOne to MongoDB.
• Added support for dynamic Keys in JSON expressions
• Improved performance of Retool's Snowflake integration
• Handle MySQL connections more reliably
• Added the option to allow dropdowns and multiselects to optionally allow itself to show out-of-range values
• Added .setImageUrl method to the Image component
• Whitelisted slack:// and mailto:// URLs for buttons and <a> tags.
• Added the ability to set a delay before initially running a query on page load.
• Added a Looker integration into Retool.
• Added magic constants USER_GOOGLE_ID and USER_PRIMARY_EMAIL that can be used in the REST API integration to pass the currently logged in user's email / google id to the API server.
• Improved how Retool serializes Firebase refs.
• Added support for all Redis commands.
• Added support for choosing a TDS version for Microsoft SQL Server.
• Fixed several issues with the Map component and it's parsing of GeoJSON
• Fixed logic for verifying if the user is currently logged into an API.
• Fixed error handling for GraphQL queries.
• Fixed a rare yet inexplicable bug where a Retool app will appear to be completely wiped clean.