Introduction

The module synchronises customer data, order data and shopping cart abandonment information between Magento and Maileon. Magento is the primary system here, which means that the customer information in Magento is considered the main information and changes to this data are synchronised with Magento. The other way around, only certain information is synchronised: the DOI confirmation and unsubscribers. For order data and shopping cart abandonments, this information is sent to Maileon as transactions (events) and can be used here either for analysis or to trigger a trigger mailing. All functions can be set via a configuration panel in Magento.

Starting up the module

Installation

The module consists of two main directories: app and lib. Under app are all configurations and the source code written for the module. Under lib is a copy of the Maileon PHP API client that can be downloaded from the developer's website1.

These files are simply inserted into the main directory of Maileon. Note: no files will be overwritten but since the module integrates into the Magento newsletter system, the class Mage_Newsletter_Model_Subscriber will be extended in the local module. If you use another module that changes the newsletter system, please deactivate it.

Note: in the Maileon account, a custom field called createdByTransaction of type boolean, as well as storeViewCode of type string must be created. Since version 0.11.2 these contact fields are created automatically if required and the process is logged in the log file ' xqueue.plugin.log '.

Configuration

When the installation is complete, you will find the new setting Xqueue Maileon-Settings in the Magento administration area under System Configuration, Figure 1 shows this option. If a 404 error message is displayed, please log out and log in to the backend again.

Open

Figure 1: Settings for Maileon

Open

Figure 2: General settings

Figure 2 shows the general settings:

• Maileon API URL
  The URL to the Maileon REST API.

• Maileon API Key
  The custom API Key².

• Maileon API Timeout
  The timeout for the connection, default (no input) is 30 seconds.

• Enable Extended Logging:
  Enable or disable logging information. If enabled, all processes are logged with Maileon, if disabled, only errors are logged.

• Print CURL Debug Data
  If this option is activated, all CURL requests on the website are printed. This allows communication errors to be analysed.

Figure 3 shows settings that are necessary for sending general transactions:

• Maileon API Key for Transactional Mails
  A custom API key that is used for sending transactional mails. This allows a separate account to be used. If the field is left blank, the general API key is used.

• Permission for Contacts Created by Transaction
  If a transaction message is sent to a contact, this contact must be created in Maileon if it does not exist. Due to the update on 2019.12.06, a newsletter permission is no longer required for sending certain emails, such as order confirmations. Therefore, NONE should be left as the permission for these contacts. However, it is also possible to set a different permission. However, caution is advised: Contacts with a permission other than NONE may be sent regular newsletters from Maileon.

Figure 4 shows the settings for regular newsletter subscriptions.

• Enabled
  Activates the synchronisation of newsletter subscribers

• Use Maileon DOI process
  If this option is activated, a DOI mailing is sent via Maileon, otherwise Magento's DOI system is used.

• DOI Mailing ID
  If more than one DOI mailing exists in the Maileon newsletter account, the key of the desired mailing can be entered here.

• Initial Permission
  When a contact is created in Maileon, it is given an initial permission status. In most cases "No Permission

• Final Permission
  By triggering the DOI process, you can choose whether a regular DOI permission or a DOI+ (DOI + permission for single user tracking) is obtained as a result of clicking on the confirmation link in the DOI email.

• Unsubscriber Check Range
  In the module configuration (config.xml) a cron job is defined that checks the unsubscribers of Maileon. This option determines the period for which this happens and should be somewhat higher than the cron job cycle so that there are overlaps. For example, if the cron job runs every 20 or 25 minutes, the unsubscribers of the last 30 minutes should be queried. Duplicate queries are processed correctly.

• DOI Hook Token
  For feedback from Maileon (DOI subscriber), a secret key must be made known in Maileon and Magento. The key can be viewed and changed here.

Figure 5 shows settings for special cases regarding registrations.

Behaviour for Customer Changing Email
When a customer changes their email in the shop, it is up to the shop operator whether the change of email is validated or not. It should be noted, however, that when the email address is changed, the DOI permission is not normally transferred to the new contact. Depending on the data protection officer, however, this is assessed differently, so the plug-in offers three settings:

Do not change subscriber

Update email in Maileon, keep permission

Update email in Maileon, remove and request new permission

Send DOI for logged in users

Permission for subscribing Customers without DOI

Subscribe Contacts when Completing an Order

Permission for Subscribing Customers when Completing an Order

Figure 6 shows the settings for order confirmations

• Enabled
  Enables the forwarding of order events to Maileon.

   

• Maileon Transaction Type ID
  The ID of the transaction type for orders in Maileon.

   

• Generate TX Type
  With this button the event can be created or queried in Maileon and entered in Magento.

  Caution: The configuration page is reloaded after pressing the button.

Custom-Field Map

Custom Options Field Map

Shadow Email

Email Override

Figure 7 shows advanced settings for orders

Send Flat Transactions

Maileon Transaction Type ID

Generate TX Type

Figure 8 shows the settings for shopping cart cancellations

Enabled

Maileon Transaction Type ID

Generate TX Type

#Of Hours Before Sending Reminder

Custom-Field Map

Custom Options Field Map

Shadow Email

Email Override

Important: The module is based on the mechanics of the newsletter system provided by Magento itself. Therefore, the settings of the standard system also apply and it is necessary to inform Magento that a confirmation (DOI) is required for new newsletter subscribers. To do this, select the menu item "Newsletter" in the configuration and set the confirmation to "Yes", Figure 9 shows the configuration.

Linking different stores to Maileon

It is possible to connect different storeviews to different newsletter accounts. Figure 10 shows that, for example, the API key and thus the corresponding Maileon account can be overwritten at storeview level. Further settings are also marked on the screenshot, e.g. it is also possible to completely deactivate the newsletter subscriber management via Maileon in certain stores.

To adjust the storeview configuration, the corresponding storeview must be selected at the top left under "Current Configuration Scope". In Figure 11, an English store was inserted as an example as a second store with its own storeview, the configuration for this was selected and the tick for "Default value" was removed. This means that a different API key can be stored for this storeview and all newsletter subscribers would be managed in a separate newsletter account.

Function of the components

Customer data

General customer data

Customers can be registered in Magento in different ways: via the registration process in general, via the registration process during an order or by a shop administrator. In standard mode (option 9 is set to "No"), customers are only created if the option for the newsletter has been clicked on.

If option 9 is set to "Yes", the contact is created in Maileon with the permission "SOI". Exception: once a customer has deliberately unsubscribed from the newsletter in Maileon, he will not be automatically registered again.

For new registrations, the hooks customer_save_before and customer_address_save_before are used in Magento. These are also called up for contact data changes. Changes from the customer or an administrator are immediately forwarded to Maileon. However, there is a special feature for newsletter subscription/unsubscription: if a customer unchecks the newsletter in the settings, he is unsubscribed; if he adds it again, he is added to Maileon with DOI+ permission without a DOI confirmation email, since he is in the Magento administration area and is therefore authenticated.

Unsubscriptions

Unsubscriptions from Magento are synchronised directly with Maileon as described. Unsubscribers who have unsubscribed from Maileon are queried by Maileon every 20 minutes by a cron job (TT_checkMaileonForUnsubscribers) in the default setting. Since customers can log in/out several times within a short period of time, for example, the change time of a customer's status is translated into the time zone of the Maileon REST API server and compared with which status is the most current. If the most recent status is a deregistration, the contact is also registered in Magento as a deregister. The time zone of the Maileon server is "Europe/Berlin".

DOI confirmations

If the Magento DOI procedure is used, the confirmation is forwarded directly to Maileon via the Maileon Rest API. If the Maileon DOI procedure is used, a webhook is called that forwards this data to Magento. In Maileon, the webhook is created under Settings Account Webhooks, see Figure 12. To do this, DOI is selected as the contact event. The POST URL is the URL that has to be called. Here is the scheme:

The "/en/" must be specified in the URL if the country extension is activated, otherwise it must be omitted. As a parameter, "email" must be specified with the contact field value "email address" and "token" with the value specified under option 10.

Saved data:

1. 'SALUTATION'

2. 'GENDER'

3. 'LOCALE'

4. 'GENDER'

5. 'FIRSTNAME'

6. 'LASTNAME'

7. 'FULLNAME'

8. 'CITY'

9. 'COUNTRY'

10. 'ADDRESS'

11. 'ZIP'

Passing custom values to Maileon

Currently, the module supports custom values for contacts only by changing the corresponding code in the file app/code/local/TT/XQueue/Helper/Data.php.
At the time of documentation, this concerns the two lines 197 and 244

In these lines, a value is passed to the Maileon custom field "createdByTransaction". Here, for example, the StoreView ID or the StoreView abbreviation could be passed.

The lines would then change as follows:

Here the module is instructed to assign the code of the current StoreView to the Maileon custom field "magentoStoreViewCode". This value can then be used to filter out all contacts of a storeview. It must be ensured that the fields used here exist in Maileon and represent the necessary data type (usually simply "text"). Please also pay attention to upper and lower case!

To do this, a function of the module can be called in the previously mentioned class. The following call creates the fields if they do not already exist:

The standard events 'magento_order', 'magento_order_product' and 'magento_abandonned_cart' can be created via the buttons in the settings. Each of these events has independent standard attributes but also a common attribute master, which can be freely assigned with additional information. The list of generic attributes is defined as follows:

• 'generic.string.1'

• 'generic.string.2'

• 'generic.string.3'

• 'generic.string.4'

• 'generic.string.5'

• 'generic.double.1'

• 'generic.double.2'

• 'generic.double.3'

• 'generic.double.4'

• 'generic.double.5'

• 'generic.integer.1'

• 'generic.integer.2'

• 'generic.integer.3'

• 'generic.integer.4'

• 'generic.integer.5'

• 'generic.boolean.1'

• 'generic.boolean.2'

• 'generic.boolean.3'

• 'generic.boolean.4'

• 'generic.boolean.5'

• 'generic.date.1'

• 'generic.date.2'

• 'generic.date.3'

• 'generic.date.4'

• 'generic.date.5'

• 'generic.json.1'

• 'generic.json.2'

• 'generic.json.3'

These attributes are not filled by default and can be filled by means of hooks from an external module.

Note: the standard attributes can also be overwritten if necessary.

Available hooks:

• xqueue_order_submitting_before

  This hook is triggered before an order confirmation transaction is sent and allows the content of the transaction to be modified.

• xqueue_order_product_submitting_before

  This is triggered before a single product of an order is sent as a transaction and allows the content of the transaction to be modified.

• xqueue_abandoned_cart_submitting_before

  This is triggered before a cart abandonment transaction is sent and allows the contents of the transaction to be modified.

A listener can register for the events in a standard-compliant way. Example:

Example of the implementation of the listener method:

Orders

Completed orders are intercepted via the Magento hook sales_order_place_after and transferred as a transaction to Maileon via the REST API.

The hooks xqueue_order_submitting_before and xqueue_order_product_submitting_before are offered on the plug-in side.

The following standard values are transferred:

• 'order.date'

• 'order.total'

• 'order.currency' (currently only €)

• 'order.productIds' (comma separated list of all IDs of the ordered products)

• 'order.categories' (comma separated list of all categories of ordered products)

• 'order.shipping_fee'

• 'order.tax_amount'

• 'order.payment_type' (not working yet)

• 'order.discount.code'

• 'order.discount.rules' (rules as JSON array)

• 'order.discount.rules_string' (comma separated list of all rule names)

• 'order.items'

• 'customer.salutation'

• 'customer.full_name'

• 'customer.first_name'

• 'customer.last_name'

• 'customer.id'

• 'billing.address.fullname'

• 'billing.address.street' (street, house number)

• 'billing.address.zip' (POSTCODE)

• 'billing.address.city' (City)

• 'billing.address.region' (state)

• 'shipping.address.fullname'

• 'shipping. address.street' (street, house number)

• 'shipping. address.zip' (POSTCODE)

• 'shipping. address.city' (City)

• 'shipping. address.region' (state)

• <Custom-Implementation-Attributes, see end of this chapter>

Furthermore, the following information is transferred for each product in the shopping cart:

• 'id'

• 'sku'

• 'name'

• 'single_price'

• 'thumbnail'

• 'image'

• 'quantity'

• 'combined_price'

• 'short_description'

• 'description'

• <Custom-Attribute>

• <Custom-Implementation-Attributes, see end of this chapter>

Maileon currently requires that a contact with the email address exists for each transaction so that the transaction can be assigned to a "contact". If a transaction is triggered for an email address that does not exist in Maileon, it will be rejected by Maileon. This behaviour will be changed in the near future but for this reason a SOI contact is currently created in this case. This contact remains even after the transaction has been sent, so that the history of sent transactions is not affected.

Note: The contacts are selected by setting the custom field createdByTransaction to "true". If these contacts are not to be sent to, a contact filter should be used which excludes these contacts. The value will be overwritten if you register for a newsletter.

Custom Attributes

Special attributes (also called custom options or custom attributes in Magento) that have been inserted into the Magento data model can be added to the products/items of the transaction by passing a mapping as shown in Figure 2 [13].

However, if this functionality is not sufficient, custom implementations can be made, e.g. collecting data from an external database and adding it to the transaction. The data can be added via the hook xqueue_order_submitting_before.

Shopping cart abandoners

Shopping cart abandoners are customers who put products in their shopping cart and then do not order them. If the customer has stored an email address, this can be used to remind them to place their shopping cart.

Two cron jobs are used for this: the first (TT_markAbandonedCarts) analyses the database and stores orphaned shopping baskets in a table (xqueue_queue). A second cron job (TT_sendAbandonedCartsEmails) reads the shopping cart abandoners from this table and creates a transaction in Maileon. The hook xqueue_abandoned_cart_submitting_before is offered to enrich the data of the transaction.

Resending a cart abandonment transaction to the same contact: only a single transaction for the contact is sent to Maileon. A re-transaction can only be triggered if the contact has either not ordered another shopping cart or has changed the contents of the shopping cart since the transaction was sent.

The following values are currently transferred:

• 'cart.date'

• 'cart.currency' (currently only €)

• 'cart.productIds' (comma separated list of all IDs of the ordered products)

• 'cart.categories' (comma separated list of all categories of ordered products)

• 'cart.total'

• 'cart.items'

• 'cart.discount.code'

• 'cart.discount.total'

• 'cart.discount.rules' (Regeln als JSON-Array)

• 'cart.discount.rules_string' (comma separated list of all rule names)

• 'customer.salutation'

• 'customer.full_name'

• 'customer.id '

• <Custom-Implementation-Attributes, see end of this chapter>

Furthermore, the following information is transferred for each product in the shopping cart:

• 'id'

• 'sku'

• 'name'

• 'single_price'

• 'thumbnail'

• 'image'

• 'quantity'

• 'combined_price'

• 'short_description'

• 'description'

• <Custom attributes, see Figure 2 [18]>

• <Custom-Implementation-Attributes, see end of this chapter>

Custom Attributes

The same options apply for cart abandonments as for orders. However, a separate hook is offered: xqueue_abandoned_cart_submitting_before.

How the shopping cart analysis works

The analysis of abandoned shopping baskets is based on 3 basic tables

1. sales_flat_quote This table comes from Magento and is used for reading to analyse which shopping baskets have a certain age.

2. xqueue_queue The job that analyses shopping baskets from table 1 writes orders for reminder emails to this table.

3. xqueue_log A second job processes the entries from Table 2. It takes the entries, generates and sends a request to Maileon, writes the request to the xqueue_log table and then deletes the entry from table 2. The job that analyses the shopping baskets compares the shopping basket ID with this table and finds out whether there is already a reminder for this shopping basket.

Overview of sent shopping cart reminders

As of version 0.13.2, the plugin in Magento offers the option under "Reports" "XQ Abandoned Carts Log" to view the entries from the database log of the shipped shopping carts and save them if required.

Sending customised transactions

The core functionality of the plugin for sending transactions can also be used by external plugins. Examples of use: Reminders of ordered products whose useful life is about to expire and which could be reordered (e.g. contact lenses) or changes to prices of observed products.

As of version 0.12.0, the transaction component of the Maileon Magento plugin consists of two areas that can be used externally:

Determining and creating transaction types based on names

Until version 0.12.0 it was necessary to create the transaction types in Maileon manually. With the update, buttons were added in the options so that the standard types (such as order confirmations) can be created in Maileon at the touch of a button and entered in the Magento configuration. The functions can also be used by other components. The class TT_XQueue_Helper_TransactionTypeHelper contains two methods for this purpose:

With the help of these methods, an extension can control the creation of transaction types. In Magento, the transaction type is saved in the configuration, but this is not absolutely necessary. The Shopware plugin asks for the name with every request and thus determines the ID. However, the consideration of performance is left to the developer, but a call normally only takes about 15ms.

The dispatch of transactions

Once the transaction type has been created in Maileon, whether via the method described under option 1, a separate API call or manually via the Maileon UI, it can be fed with data. It is important to note the change in the Maileon update of 2019.12.06:

• Previously, every contact that was to be sent a transaction had to have a valid permission. This leads to conflicts when sending transactions and newsletters in the same account.

• After the update, it is possible to mark trigger emails as "advertising-free", such as order confirmations. These transaction emails can then be sent without permission.

This update eliminates certain design decisions that were previously implemented.

The general helper class TT_XQueue_Helper_Data contains a method "sendTransaction" for sending, which regulates the sending of a transaction message.

The use can best be shown with the example of order confirmations. First, the event is reacted to in the corresponding observer TT_XQueue_Model_Observer in method placeOrder by creating a transaction event object. Exemplary code excerpt:

The method sendOrderTransaction is a wrapper that forwards the data to sendtransaction and adds a few convenience methods, such as sending a copy to a second contact. These functions are optional.

FAQ

Q: If I have not logged in as a customer but want to log in with a customer's email address, I get an error message "Signup for the newsletter is not working, another user is using the same email address". Why is this and how can I turn it off?

A: This behaviour comes from Magento itself. The corresponding code is in the class app\code\core\Mage\Newsletter\controllers\SubscriberController.php, around line 60. To change the behaviour, the class must be overwritten, and the case can simply be commented out.

Q: I use OneStepCheckout and when I sign up for the newsletter in the order process a customer gets 2 DOI mails, why?

A: OneStepCheckout has one call for a DOI while processing the order data and one which registers customers after processing. However, due to a bug in some versions, the two cases are not mutually exclusive, so that a customer gets both DOIs. The problem has been solved so far by overwriting the class app/code/local/Idev/OneStepCheckout/Block/Checkout. In the code section responsible for login registrations, the flag was then set to false, which triggers the second registration.

Changelog

0.13.2 - 2020.01.27

• Added admin view and components for cart abandonment logs

• Refactored file structures (models, collections, ...)

• Cleaned code

0.13.1 - 2020.01.23

• Fixed bug in naming scheme of controllers for "create transaction" buttons

• Changes in transaction type for cart abandonment events:

  • Added attributes cart.discount.code (string), cart.discount.total (double), cart.discount.rules (json, containing names of rules), cart.discount.rules_string (string, comma separated list)

• Changes in transaction type for cart abandonment events:

  • Renamed attributes order.voucher.code to order.discount.code and order.voucher.total to order.discount.total

  • Added order.discount.rules (json, containing names of rules) and order.discount.rules_string (string, comma separated list)

• Added to all transaction types generic strings (5), integer (5), double (5), date and
  JSON (3)

0.13.0 - 2019 .12.05

• Changes in naming for order events:

  • Renamed order.voucher_code to order.voucher.code

  • Added order.voucher.total (type double)

• Added filling voucher code and total

0.12.0 – 2019.11.29

• Added optional API-Key for transactions (if no key is provided, the regular API key will be used

• Added option for setting the permission of contacts created by transactions

• Added option for deciding whether to reset the permission of contacts created by transactions

• Added option for subscribing customers on orders automatically, if no active unsubscribers

  • For customers only

  • For guests and customers

• Added option for setting the permission of contacts that are subscribed with orders, automatically

• For orders: added flat transaction bearing a list of categories (one transaction for each category)

• Changes in naming for order events:

  • billing_address is renamed to billing.address, attributes are named more readable, now

  • shipping_address is renamed to shipping.address, attributes are named more readable, now

  • Added customer.first_name and customer.last_name

  • Renamed list of product IDs from generic_fields.string1 to order.productIds

  • Renamed list of categories from generic_fields.string2 toorder.categories

  • Renamed payment_type to order.payment_type

  • Renamed voucher_code to order.voucher_code

• Changes in naming for cart abandonment events:

  • all fields with "order" to "cart", e.g. order.items was renamed to cart.items

  • Renamed list of product IDs from generic_fields.string1 to cart.productIds

  • Renamed list of categories from generic_fields.string2 to cart.categories

• Added method for automatically creating transaction types

• Added option for handling subscribers on changing the e-mail address

  • Keep subscriber address unchanged

  • Update Maileon contact with new e-mail address but keep permission

  • Update Maileon contact but remove permission and send DOI mail

¹ http://dev.maileon.com/maileon-php-api-client

² http://dev.maileon.com/api/rest-api-1-0/securitymechanism