Data import

Product import

Scan & Go is mainly powered by the data provided through the DataHub import jobs on the Management Console. The first steps in the Scan & Go customer journey primarily use this data. In this article, we will focus on the product import and how this import is essential to scan products on the customer app.

The product import is one of the available import jobs out-of-the-box in the shopreme DataHub. It can be set up in the Management Console under the Jobs section. It imports products “globally” for the whole system; in other words, uploading products for specific stores only is not supported. However, pricing can be store-specific by adding a storeId to the storePrices array.

This is a quick overview of where the product import is most relevant during the Scan & Go journey:

We understand that an international retailer would have a different logic for the product information, pricing, and tax details. Because of that, for a multi-country setup, shopreme provides a separate server for each country.

Required fields

The two required fields in the product import so that a product can be imported and visualized on the Management Console are productNumber and productName. These two fields should be consistently the same for all products in one server (i.e., in the same country or fiscal region).

The productNumber identifies the specific product entry and be used to update the same product information in subsequent imports. It should also be recognizable by the cash register (if an external system is implemented). The productNumber, in combination with the unitCode, must be unique. This means that, for example, the productNumber can be imported twice only if the unitCode can distinguish the two products from each other (e.g., the unitCode could be “ST” for a Single item or “PK” for a Pack).

The productName is the display name that the customer will see on the Scan & Go app.

However, to be able to scan a product on the Scan & Go customer app right after running the import jobs, it is necessary to include information in the scanCode field. The data type is an array of strings: it is possible that the same product can be purchased by scanning more than one barcode. In that case, all allowed barcodes for that product should be provided.

All product import fields

In the shopreme DataHub section of the Management Console, you can find some JSON examples and documentation that explains in more detail what each field represents and what the resulting JSON file would look like. You can check this under DataHub > Jobs > find the relevant job type, and then click “Edit” on the Job type. A new menu will be displayed, where you can click “Documentation” – there you will see the JSON example and its documentation.

All the available fields in the shopreme import are the following:

Field name & data type	Parent field	Description
`additionalFields` Object		Additional information that can be added to the product
`externalInformationSDK` String	`additionalFields`	Maximum character limit: 10000 (Excess characters will be truncated) The field can be used to propagate information to client devices. Information within this field should not contain secrets and is limited to 10000 characters. If more information than just a single value is needed, storing a base64-encoded JSON object is recommended to avoid escaping issues.
`externalInformationDataHub` String	`additionalFields`	Maximum character limit: 10000 (Excess characters will be truncated) This field can be used to propagate information to third-party systems that communicate with the DataHub API. Information stored in this field will only be propagated to users of the DataHub API and not to client devices. It may contain secret information. The field length is limited to 10000 characters. If more information than just a single value is needed, it is recommended to store a base64-encoded JSON Object to avoid escaping issues.
`productNumber` String Required		Maximum character limit: 255 (Excess characters will be truncated) Unique identifier of the product. This field will be used to identify the product in all DataHub interfaces when communicating with third-party systems. This identifier must be used to update product information in subsequent imports.
`unitInformation` Object		A combination of code and quantity that reflects the packaging unit for the product. Setting this field is usually not necessary if the `productNumber`s are unique without any further unit information.
`unitCode` String Required	`unitInformation`	Maximum character limit: 16 The packaging unit of the product, usually in the form of a short code like ‘ST’ (piece), ‘TR’ (tray), ‘PL’ (pallet). The combination of productNumber and unitCode uniquely identifies a product, which means that multiple products can share the same `productNumber` given that they all have different units. This field has no relation to the number of products in this packaging unit, which is instead expressed by the unitQuantity field. The unitCode is limited to 16 characters.
`unitQuantity` Integer Required	`unitInformation`	The quantity for the given unit. For example this should contain 6 for a six pack of drinks.
`productName` String Required		Maximum character limit: 255 (Excess characters will be truncated) The name of the product, as shown on the customer app.
`description` String		Maximum character limit: 10000 (Excess characters will be truncated) Detailed description of the product that may contain information such as nutritional details, instructions, or product origin. The description allows a few whitelisted HTML tags: `b`, `em`, `i`, `strong`, `u`, `ol`, `ul`, `li`, `strike`. Other tags will be filtered out.
`unitDescription` String		Maximum character limit: 255 (Excess characters will be truncated). Human readable description of the unit the product is sold in – e.g., “0.7l bottle”, “300g can”, or “24 pieces”.
`ageRestricted` Integer		Default value: null. Age restriction of the product in years. If 0 or null, the product is not age-restricted. Only non-negative integer values are valid.
`theftProtectionDevice` Enum: [NONE, GENERIC]		Default value: NONE. Flag that indicates if the product has a theft prevention device.
`scanCodes` Array<ScanCode>		List of all the `scanCode` that can be used to scan this product. If additional information (e.g., weight or price) is encoded in some `scanCode`, the format of these “template codes” will be configured on the shopreme middleware – the format is not part of the import. If you want to use such template codes, please contact your shopreme integration consultant.
`scanCode` String Required	`scanCodes`	Maximum character limit: 255 (Excess characters will be truncated) Content of the `scanCode`. If the same `scanCode` is set for more than one product, a product chooser will be shown in the customer app. This can be useful if the user should be asked to choose between different packaging sizes when scanning a particular product (e.g., “single bottle” vs “6-pack”).
`type` String	`scanCodes`	Maximum character limit: 255 (Excess characters will be truncated) This field is reserved for customer-specific integration. It has no effect by default. This property can be used to give special meaning to a `scanCode`, which might be useful in case shopreme needs to perform special actions on these `scanCode`. Please contact your shopreme integration consultant in case this field is necessary.
`emptyTemplateBehavior` Enum: [SHOW_ERROR, ADD_ONE, MANUAL_INPUT]	`scanCodes`	This field defines what should happen when a template scancode is scanned with the ‘unit’-part being empty. Templates are scancodes that contain encoded information (e.g., weight or price). Such templates usually consist of a product identifier and unit information. Example: 24 12345 06789 1 24: Prefix to identify the specific template (“weight” in this case) 12345: Product identifier, imported into shopreme as a scanCode 06789: The unit information, i.e. 6789 g = 6.789 kg 1: Checksum This field specifically defines the behavior if the ‘unit’-part of the scancode is all 0 (for the example above, that would be “12 3456 000 1”). Available values: `MANUAL_INPUT`: Opens a manual input that allows customers to enter the unit value directly. `ADD_ONE`: Adds one unit (e.g., “1 kg”) of the product to the basket. `SHOW_ERROR`: Shows an error message to the customer, and the product is not added to the basket.
`scanError` Object	`scanCodes`	If this field is set, an error occurs when a product is scanned with this `scanCode`. The product will not be added to the basket when scanning this `scanCode`. The error `message` that is displayed can be customized and translated in the customer app using the `code` field, which will be passed along to the shopreme mobile SDK. It is important to note that this `scanError` is only shown if a valid product is scanned with this `code`. If, for example, a product without a price is scanned this way, it would lead to an error before providing the scanError’s `message`. Example Usage: A product should be put on a scale rather than being directly scanned by the customer. However, the product’s PLU is printed on some shelf labels to aid employees when stocking the store. In this case, the PLU can be imported with a `scanError` that tells the customer to put the product on the scale first.
`code` String Required	`scanError`	Maximum character limit: 255 (Excess characters will be truncated) The technical code for this error, e.g., “SCANCODE_INVALID”. This field will never be shown to a customer directly. The code will be passed along to the shopreme mobile SDK, where it can be used to identify this specific error. The mobile app team can then provide a custom error message for the error in case the message property is not sufficient.
`message` String Required	`scanError`	Maximum character limit: 255 (Excess characters will be truncated) This error message will be shown to the customer if the mobile app does not implement any custom error handling for the accompanying `code`. Text is limited to 255 characters.
`image` Object		Default value: null. Each product may have one image that is displayed to the customer when it is scanned or appears in a search result. There are three different ways to import the product image. Exactly one of the three fields should be set. If multiple fields are set, the field with the highest priority will be used. If none are set, a warning will be logged. If the field is not provided or null, existing values will be cleared.
`name` String	`image`	Name of the image that was previously imported into the shopreme system, in the Media section. Priority: tertiary
`imageUrl` String	`image`	URL to the image. shopreme will download the image once during the first import, and the image will be stored and served via the built-in shopreme CDN. The image will never be updated as long as its URL does not change. Priority: secondary
`imageCdnUrl` String	`image`	URL to the image on an external CDN. The image will not be downloaded in the import but directly served to the customer app. The customer app will request the image directly from the CDN. Priority: primary
`canManuallyIncreaseQuantity` boolean		Default value: true. This flag defines whether the customer can use the app’s quantity buttons to increase the amount after scanning. If set to false, the customer needs to scan each individual item.
`maxQuantity` Integer		Default value: 2147483647. Defines how many units of the product can be purchased in a single basket.
`deposits` Array<Object>		Default value: null. Deposits are handled like normal products in the shopreme system. If a product contains deposits, this field is used to reference the corresponding deposit products. Important note: Deposit products have to be imported before the products that reference them in this field. Example: A crate of beer includes two deposit items: one for the crate itself and six for the individual bottles. This results in two deposit product entries: one with a quantity of 1 (for the crate) and another with a quantity of 6 (for the bottles).
`productNumber` String Required	`deposits`	Maximum character limit: 255 (Excess characters will be truncated). Product Number of the referenced deposit product. Deposit products are not allowed to be available with a different `unitCode`. Only a single unit or none at all is allowed
`quantity` Integer Required	`deposits`	Quantity of the referenced `deposit` product. Defines how many times the `deposit` product will be added after 1 item of the source product was scanned: e.g., “6 x bottle” for a 6-pack of a drink.
`spotCheckBias` Double		Default value: 0.00. Increases or decreases the likelihood of a customer being spot-checked when their basket contains this product. Values should range between [-1, 1] with precision to two decimal places, where 1.0 equals a 100% increase in the chance of being spot-checked and -1 meaning a 100% decreased chance of being spot-checked.The value will be rounded to two decimal places. If not provided, the default value is 0.00.
`categoryIds` Array<String>		Default value: null. The list of category IDs (e.g., [“extCat-fruitsAndVegetables”] ) this product belongs to. If the field is not provided or null existing values will not be updated. To remove previously assigned categories, set an empty array as a value here.
`userSelectableDiscountStickers` Array<Object>		Default value: null. Defines if (and which) discounts the customer should be allowed to manually apply to this product. For example, this property can be set for products that might have discount stickers on them.
`customerFacingName` String Required	`userSelectableDiscountStickers`	The name of the discount, as seen by the customer in the app. It should be short and unique, for example: “-1€”, “-20%”
`externalId` String Required	`userSelectableDiscountStickers`	The identifier used by the external pricing system to correctly apply the promotion to the basket if the manual discount is added to the cart.
`discountClass` String Required	`userSelectableDiscountStickers`	This value is passed through to the shopreme mobile SDK and has no semantic meaning to shopreme. The mobile app team using the shopreme SDK can use this value to adjust the style of the manual discount.
`image` Object	`userSelectableDiscountStickers`	Default value: null. Images for discount stickers are currently unsupported. Each `discountSticker` may have an image that will be shown on the discount selection UI for the customer, if available. There are three different ways how to import the image. Exactly one of the three fields should be set. If multiple fields are set, the field with the highest priority will be used. If none are set, a warning will be logged. If no image is set, only the customerFacingName will be displayed in the UI.
`searchOptions` Object		Default value: null. This object contains whether and how products should be displayed as a search result.
`showInShoppingListSearch` boolean	`searchOptions`	Default value: true. Indicates whether the product should be displayed in the shopping list search. (e.g., internal products such as a `deposit` or crates should have this property set to false).
`showInProductScannerSearch` boolean	`searchOptions`	Default value: false. This value indicates whether the product should be displayed in the search accessible from the scanner, which allows adding the search results to the cart without scanning them. Only products without a `scanCode` should be displayed here.
`boostFactor` Double	`searchOptions`	Default value: 1.00. This field can be used to prioritize the product in search. The higher the `boostFactor`, the more privileged the product will be as a search result. The scale ranges from 1 to 10, and it will be rounded to two decimal places, 1 being the default and does not change how the search will handle the product, and 10 being the strongest possible boost.
`additionalSearchTerms` Array<String>	`searchOptions`	Default value: []. Specifies additional search keywords that can be used to find this product. By default, the `productName`, the names of categories the product belongs to, its `description`, and the product’s `scanCodes` can be used to find the product.

Basic product import

The term “basic product” refers to a simple item with a unique ID, a unique barcode, no special characteristics, and no special requirements. When a customer scans such type of product, a single item will be added to the cart without additional steps or settings to configure. Here is an example of what a basic product looks like in a JSON import file:

{
    "productNumber": "00-000002",
    "unitInformation": {
      "unitCode": "STK",
      "unitQuantity": 1
    },
    "productName": "Butter croissant",
    "description": "A very fine croissant. French style. Fresh from the oven!. It might also contain <i>simple</i> <b>markup</b> but not too much.",
    "unitDescription": "50g unit",
    "ageRestricted": 0,
    "theftProtectionDevice": "NONE",
    "scanCodes": [
      {
        "scanCode": "9564500197228",
        "type": "customer_property",
        "scanError": null,
        "emptyTemplateBehavior": null
      },
      {
        "scanCode": "00000",
        "type": "customer_property",
        "scanError": {
          "code": "INVALID_EAN_CODE",
          "message": "Invalid ean code for this croissant"
        },
        "emptyTemplateBehavior": null
      }
    ],
    "image": {
      "name": null,
      "imageUrl": "https://i.imgur.com/fi31098.png",
      "imageCdnUrl": null
    },
    "maxQuantity": 0,
    "canManuallyIncreaseQuantity": true,
    "deposits": null,
    "spotCheckBias": 0.1,
    "categoryIds": [],
    "userSelectableDiscountStickers": [],
    "searchOptions": {
      "showInShoppingListSearch": true,
      "showInProductScannerSearch": false,
      "boostFactor": 0,
      "additionalSearchTerms": [
        "Bakery"
      ]
    },
    "additionalFields": null
  }

Once this product is uploaded to the Management Console through the product import job, it will be available to view, as in this example:

As soon as the product is available in the Management Console and has a barcode and a price, it can already be scanned by customers using Scan & Go (assuming the cart evaluation is done by shopreme and no further adjustments are necessary on the cash-register side to be able to process the final price of the product). Here is an example of what the imported data will look like on the end-user side:

Product import data in scanner overlay

Product import data in product detail page

Special EAN products

In addition to basic products, some products are not simply sold per unit. For example, fruits and vegetables are sold by weight, and some retailers may sell arts-and-crafts materials by length or area. To allow customers to buy such products with Scan & Go, retailers should generate “special EANs” that follow a regular expression (regex) and contain a PLU (“Price look-up code”).

When it comes to importing these products, a product sold in fractional quantities (e.g., per kilogram) will include the PLU as a string in the scanCode array. It is also recommended to add a string in the pricePerUnit for a user-friendly description.

Here is a simple example of a product imported with a PLU, which will need to be sold by using fractional quantities:

{
	"productNumber": "00-000003",
	"unitInformation": null,
	"productName": "Fresh tomatoes",
	"description": "Fresh tomatoes sold by kilo. Fresh from the local farmers.",
	"unitDescription": "Per kilo",
	"ageRestricted": null,
	"theftProtectionDevice": "NONE",
	"scanCodes": [{
		"scanCode": "98331"
	}],
	"storePrices": [{
		"price": 129,
		"strikePrice": 0,
		"taxRate": 1900,
		"storeIds": null,
		"pricePerUnit": "1,29 €/kilo"
	}],
	"image": {
		"name": "ImageIdentifierOfAPreviouslyUploadedImage",
		"imageUrl": null,
		"imageCdnUrl": "https://i.imgur.com/zK59F7Z.png"
	},
	"canManuallyIncreaseQuantity": false,
	"categoryIds": [
		"garden-decoration"
	],
	"availableManualDiscounts": [],
	"searchOptions": {
		"showInShoppingListSearch": true,
		"showInProductScannerSearch": false,
		"additionalSearchTerms": []
	}
}

Once this product has been imported, it will be available in the Management Console, with a corresponding price and price per unit (only to display to customers in the app). The imported PLU will appear in the list of barcodes.

Product with PLU in the Management Console view

Depending on the business needs, different rules can be set up during the integration process. The underlying rules and regular expressions can only be configured by the shopreme development team. Please consult your shopreme integration consultant if you need to implement a change to the scan code logic.

The patterns to recognize different types of barcodes cannot be modified on the Management Console. However, the most common format is generally structured following this pattern:

An EAN with variable weight: XX-PPPPP-GGGGG-C

X (2 digits): A prefix that determines the unit, e.g., centimeters, square centimeters, grams, etc. This prefix is configured on the shopreme backend during the integration stage.

P (5 digits): The product PLU, which is padded with leading zeroes if it is shorter than five digits.

G (5 digits): The weight in grams.

C (1 digit): A control number.

For our sample product above, the following special EAN with encoded weight could be generated: 2298331022501. The barcode would then be interpreted as follows:

22	This number signals that the barcode corresponds to a product that is sold per weight. The default unit is grams, but the customer will see it as kilograms.
98331	The PLU for this product, as provided in the product import JSON file, in the scanCode array.
02250	The weight of the product in grams: 2.25 Kg.
1	A control number.

To complete the example, when customers scan this special barcode with encoded information, they will see the correct weight, product name, and price. Moreover, the “increase quantity” button is disabled.

Weight-encoded barcode in scanner overlay

Weight-encoded code – Product description

Cart evaluation of special EANs

For these special EANs with encoded information, shopreme sends the actual scanned barcode information to the /evaluate endpoint instead of the imported PLU. The unit information in the barcode is also sent in the field salesUnitPerPiece. This allows any external pricing engine to parse the barcode and calculate the correct price. The external system can then take the information from the barcode or the salesUnitPerPiece field to calculate the unit price.

Offline product export

For shoppers to scan products as quickly as possible and avoid loading times, shopreme Scan & Go downloads an offline product export to the shopper’s app. This export file is downloaded the first time the app is opened and updated once a day (only if the Scan & Go app is opened again).

In other words, this feature allows that when a shopper scans a product, it is first looked up on the local offline product export. If it is not there, it will be searched on the product database on the shopreme server.

What products are included in the offline export?

Only products that are imported with default prices in the import file are included in the offline export. Alternatively, the products should have a price for every store that is included in the export file.

This means that weighed products, among others with special EANs, cannot be included because the scan codes are variable.

Data import ›

Search...

⌘K

Support

Can’t find the answer to a question or want to learn more about a feature?

Search...

⌘K

Support

Can’t find the answer to a question or want to learn more about a feature?

Get in touch