In this article, you’ll learn the difference between logical and physical joins, the factors SQL Server considers when deciding how to execute a join, and two approaches to resolving a slow SQL query.

The Query

Here’s the SQL query that was running slowly. I’ve changed the table names and column names to obscure details about my client. After 5 minutes of running the query without finishing, I decided to abort it and troubleshoot it.

SELECT
    v.CustomerID,
    COUNT(v.ID) AS NumberOfVisits
FROM Customers AS c
INNER JOIN Visits AS v ON c.ID = v.CustomerID
WHERE v.VisitDate BETWEEN '2024-01-01' AND '2024-02-01'
GROUP BY v.CustomerID

It’s a simple query: it just joins customers to visits and selects the number of visits that each customer made within the date range specified.

It’s important to note that the customers table is tiny, about 4,000 rows, and the visits table is large, several hundred million rows, so it’s understandable for this query to be performance-intensive. This exact query performed very fast in staging, but there was very little data for that date range in staging, so it makes sense for it to perform differently.

What really confused me is that this query was a glorified copy-paste from a query that we had been using in production to run ad-hoc reports. When we ran it, it was dealing with production-scale data, so why did this version perform so poorly?

The Execution Plan

The first step to troubleshooting a slow SQL query is to get an actual execution plan (if you can). This will show you exactly what steps SQL Server took to execute your query, and runtime stats such as how much time and CPU each step took, how many rows were returned by each step, etc.

In SQL Server Management Studio (SSMS), press CTRL+M, then run your query. By default, this shows the execution plan only after the query finishes, which can be an issue for very slow queries. If you don’t want to wait, you can press CTRL + M and click the button near the top of SSMS labeled “Include Live Query Statistics”. This not only shows you the execution plan right away, but it also shows you the runtime stats (rows read, elapsed time of each step, etc.) in real time, so it updates as you watch it.

I can’t share the screenshot of the execution plan from production, but here’s what it looked like in a development environment when I ran it later. Table and column names have been changed to obscure client details.

I was profoundly surprised by this execution plan because there’s no physical join operator, despite the fact that there’s a logical join in the SQL query.

When you go to a restaurant and order steak au poivre, you don’t give the waiter instructions to cook it (“crush some peppercorns, pat the steak dry, set the grill to medium-high…”). You just ask for steak au poivre, maybe specify medium-rare, and the chef makes it happen.

SQL is the same way. SQL is a declarative language. When we write a SELECT query, we’re not writing instructions for how to get the data; we’re declaring what data we want. SQL Server needs to make decisions about how to execute it, such as which indexes to use or whether to go parallel.

Similarly, when we write any kind of JOIN in a SQL query, what we’re writing is a logical join. Examples include INNER JOIN, LEFT OUTER JOIN, RIGHT OUTER JOIN, CROSS JOIN. SQL Server needs to figure out how to make your join happen, and it will do that by using one of several physical join operators. The most common physical join operators are Nested Loops, Hash Match, and Merge Join, so every execution plan you look at for queries involving logical joins will include at least one of those three operators.

When a Logical Join Does Not Require a Physical Join

Our slow SQL query is an exception to that rule. Our query clearly has a logical join (INNER JOIN) between two tables, but the SQL Server chooses an execution plan with no physical joins.

Why?

The first thing that came to mind is that the SQL query, despite joining the Customers table and the Visits table, only includes fields from the Visits table in the SELECT clause, WHERE clause, and GROUP BY clause. We’re not using any fields from the Customers table other than its ID field as the join key.

That’s part of the explanation, but it doesn’t explain all of it. The whole point of an INNER JOIN is to return only the records that have matches in both tables. So, even though we’re only SELECTing fields from one table, SQL Server still has to determine whether a given key value is present in both tables. How can it do that without performing a physical join?

The answer is the foreign key reference that enforces the relationship between these two tables. Specifically, we have a foreign key on the Visits table on the CustomerID field, referencing the ID field of the Customers table. This foreign key enforces that no record can exist in the Visits table without a corresponding record in the Customers table.

So, let’s look at this from SQL Server’s perspective:

1. The user is asking me to perform an INNER JOIN.
2. He’s asking me to filter and group based on fields from only one of the tables.
3. He’s only asking for fields from that same table.
4. I know for a fact that all records in Visits have a corresponding Customer record.
5. Why the heck would I bother with a physical join? I’ll just read from the Visits table and ignore Customers.

Physical joins can be memory-intensive and time-consuming. Generally, if we can avoid a physical join, we’ll get our results faster. So, the execution plan makes sense.

This isn’t just theory, either. I tested it in a development environment by removing the foreign key and running the SQL query. As expected, the execution plan included a physical join. Adding the foreign key constraint back to that field and running the query again resulted once more in an execution plan with no joins.

Why Is the Query Slow If We’re Avoiding a Join?

Now that we’ve identified why the execution plan has no physical join, let’s circle back to the problem at hand: the query is slow. Why is it slow, and how do we fix it?

Let’s take another look at the execution plan, specifically the first step.

SQL Server is choosing to do an index scan. That means it starts at the very “top” of the index and reads through it to the end. It chose this index, “IX_Visits_CustomerID_VisitDate,” because it includes all the fields our query needs. This index contains a couple of fields of each row, which means SQL Server can fit more rows on a single page of data, allowing it to read more rows per page.

However, the order of the keys in this index is hurting performance. CustomerID is the first key, and VisitDate is the second key. That means the rows in the index are sorted first by CustomerID, then by VisitDate. If it was the other way around, all the rows that match our WHERE clause would be right next to each other in the index and SQL Server would be able to do a “seek” to read only the minimal pages it needs to get the data we’re asking for.

Solution 1: Add a New Index

The most obvious solution is to add a new index with VisitDate as the first key and CustomerID as the second. Here’s the resulting execution plan:

Even in this development environment, where the data is much smaller, this resulted in a 100x speed boost—the query finished in about 0.1 seconds instead of 10 seconds.

However, indexes aren’t free, especially on large tables. The more indexes you have on a table, the longer it takes to write to that table when rows are updated or inserted. This table already has more indexes than I’d like, so I’d prefer to avoid adding a new one.

Additionally, this query runs once a month as part of an automated report, so it feels silly to add a whole index just to make this one query faster.

When we solve a slow query by adding, modifying, or removing indexes, that’s called “index tuning”.

Solution 2: Change the Query

In general, if you can make a query faster by rewriting the query rather than adding indexes, you should do so. I ran the old version of the query that was previously used to generate the ad-hoc reports and found that it used a physical join operator. The reason was simple: the original query joined the Visits table to a temp table representing Customers rather than the table itself. The Visits table obviously did not have a foreign key reference to this temp table, so it was forced to perform a physical join.

For the new version of the query, I forced a physical join by selecting a field from the Customers table instead of the Visits table:

SELECT
    c.ID AS CustomerID,
    COUNT(v.ID) AS NumberOfVisits
FROM Customers AS c
INNER JOIN Visits AS v ON c.ID = v.CustomerID
WHERE v.VisitDate BETWEEN '2024-01-01' AND '2024-02-01'
GROUP BY c.ID

In this version of the query, I’m using the Customers table’s ID field for both the GROUP BY and one of the fields in the SELECT, while still using fields from the Visits table in the WHERE clause and the SELECT clause. This forces SQL Server to do a physical join, and the execution plan looks like this:

In my development environment, the query takes 0.455. In production, it takes a couple of seconds, which is fast enough for our purposes. It runs as part of an automated report and must complete within 30 seconds to avoid timeouts.

Let’s zoom in on that physical join operator:

SQL Server chose the Nested Loops join. This physical join operator is very fast and efficient when you have a small result set on one side of the join compared to the result set on the other side. We have just a few thousand rows in our Customers table and a couple hundred million rows in our Visits table, so Nested Loops is the right choice.

The way this join works is that SQL Server loops through the smaller data set, the Customers table, and for each row, performs a seek on the Visits table to get the rows corresponding to that CustomerID value. These seeks are fast because CustomerID is the first key in the index.

Because this version of the query meets our performance needs without requiring a change to the database’s structure (e.g., by adding an index), it’s the solution we chose.

When we solve a slow query by rewriting or tweaking the query text itself, as we did here, that’s called “query tuning”.

The Takeaway

What have we learned? We can take away the following lessons about SQL Server:

1. SQL Server does not always need to perform a physical join to execute a logical join.
2. Foreign key references, or lack thereof, can impact whether SQL Server performs a physical join.
3. If you only use fields from one table in your SELECT, WHERE, ORDER BY, or GROUP BY clauses, SQL Server might not bother with a physical join.
4. When trying to speed up a slow query, you should first try to rewrite the query before resorting to adding or changing indexes.
5. Sometimes an unusual solution is the best one. In rare cases, adding a join can improve performance.

This was a deep dive into a fascinating detail of how SQL Server decides when to do a physical join and how that impacts performance.

If you rely on SQL Server or Azure SQL for your business-critical applications, you need database experts who can calmly work under pressure to solve tricky problems like the one described in this article.

The post SQL Server Performance Tuning: When a Join Isn’t a Join appeared first on DMC, Inc..

1. Download and Installation
2. Adding the Database Project to a TwinCAT 3 Solutions
3. Writing Code to Write to the Database Server
4. Testing the Database Connection

Being able to store long term data is one of the most crucial parts of a manufacturing or testing environment. Having access to this data can allow a team to track long term trends of their systems. For instance, one team could track the uptimes of the various production lines to find where the bottlenecks in the process flow are. Another team could use their motor’s speed data to track when it needs maintenance.

Beckhoff TwinCAT 3 offers the ability to run a database server directly on the IPC using their TF6420 module. This module allows you to send data up to most types of databases, though this blog will provide specific examples by using a PostgreSQL database since it is free to use, open source, and uses structured query language (SQL).

1. Download and Installation

1.1. Download

First, Download TF6420 from the Beckhoff website. The installer is located in Documentation and Downloads > Software and Tools.

1.2. Installation

Once downloaded you will need to run the .EXE on both the engineering computer and on the PLC. To download onto the PLC, the simplest way is to move the .EXE onto a USB and plug this into the PLC. Once the USB is plugged in, you can either remote into the PLC (Remote Desktop Connection) or connect a monitor and mouse using any of the outputs on the PLC. Once connected, run the .EXE on the PLC and install the software.

1.3. Open TwinCAT XAE

Once the installer is complete, open the TwinCAT XAE solution in which the database connection should be installed. If you haven’t already installed TwinCAT XAE or configured your Beckhoff IPC yet, check out our setup blog on how to get started and then come back to step 2.

2. Adding the Database Project to a TwinCAT 3 Solution

2.1. Add New Database Server Project

Within the project, right-click on the project solution -> Add -> New Project

In the window that opens, select TwinCAT Connectivity and then choose “Empty TwinCAT Database Server Project”

2.2. Add and Configure the Database

Once the new project is added, there should be a default TcDbServer already added under the project. There is currently no database in the project, but this can be done by right-clicking TcDbServer -> Add New Database

You will then see a new DB server appear under TcDbServer. The server can be configured by double-clicking the DB. This will bring up the configuration window. Currently the window is empty, but this can be fixed by selecting the database type pictured below. For this example, a PostgreSQL database will be used.

Once the database type is selected, the settings will appear and can be configured there. For PostgreSQL, the first few settings are Host, Database, and Port. The next option is Authentication which contains a drop down with different kinds of authentication. The example below is going to use Username/Password.

Note: The password used here is encrypted and is not stored in plain text anywhere on the PLC

2.3. Set Target AMS Net ID

Once all the information is added, the connection is not quite ready to be tested. The AMS Net ID must be set up first. To do this, go back to TcDbServer and double click the tab. Once the server settings window is open, there is a dropdown menu in the top left where all the options for Ams Net Ids appear.

For testing on a local machine select <Local>.

If this is running on a PLC and you are connected to the PLC, the name of the PLC will be on this menu and you can choose that instead.

2.4. TF6420 License

Before checking the connection to the server, the TF6420 license must be activated. For now, a trial license can be generated. First start by going to the Manage License tab at the top of the window that has opened. Next scroll all the way down to TF6420 TC3 Database Server and check the box next to it that says cpu license.

After this box is checked, go back to the main license screen and into the Order Information (Runtime) tab. Once here go down to the 7 Days Trial License and click on that. A small screen will then appear in which you will type a short captcha. Once this is typed, hit Ok and the license will be generated.

2.5. Activate Database Configuration

Next, activate the configuration of the database connection by right-clicking the DB and selecting Activate Configuration.

(Optional) The database server toolbar is optional, but will be beneficial moving forward. At the top of the screen on the task bar, right-click and select TwinCAT Database Server. This will add a new set of tools at the top that can be used with the databases (one of them being activate configuration).

Here is what the toolbar will look like when added:

2.6. Check Database Connection

After activating configuration, check this connection by going back to the DB page. At the bottom, there will be 2 options, CREATE and CHECK. Select CHECK.

If everything went well, you should see a pop up that says the check succeeded. If not, verify all database information is correct and go back to step 8.

2.7. Add Database to Configuration Pool

Once the database is configured, it needs to be added to the DB configuration pool. This can be done by right-clicking on the DB and hitting the Add to DB Configuration Pool tab.

3. Writing Code to Write to the Database Server

3.1. Installing Reference Libraries

Once added to the DB Configuration Pool, there are a few basic functions to test the connection and write a few queries. In order to use these, we will need to install 2 additional libraries. To do this, go to the PLC project that the database will be called in. Then go to the references tab in the project tree pictured below. Right-click this and then select Add library.

This will open a library window where the 2 libraries needed for database are located. Tc3_Database and Tc3_EventLogger.

3.2. Example Code

We can now use the built-in TwinCAT functions to send a basic SQL command. Again, for this example we’re using PostgreSQL functions and syntax. Under POUs, open MAIN and put the following code in the variable declaration section:

VAR 
    nStep, nNextStep : INT; 
    sQuery : STRING; 
    fbDatabase : FB_SQLDatabaseEvt(sNetID := '', tTimeout := T#5S); 
    fbSqlCommand : FB_SQLCommandEvt(sNetID := '', tTimeout := T#5S); 
    tcMessage : I_TcMessage;  
END_VAR

VAR 
    nStep, nNextStep : INT; 
    sQuery : STRING; 
    fbDatabase : FB_SQLDatabaseEvt(sNetID := '', tTimeout := T#5S); 
    fbSqlCommand : FB_SQLCommandEvt(sNetID := '', tTimeout := T#5S); 
    tcMessage : I_TcMessage;  
END_VAR

Here is a brief explanation of each variable:

• nStep and nNextStep are used to drive a state machine.
• sQuery is the raw string for the query we want to send to the database
• FB_SQLDatabaseEvt is responsible for connecting, disconnecting and overall communication with the database.
• FB_SQLCommandEvt is responsible for generating a SQL command to be sent via the FB_SQLDatabaseEvt. Note: A standard STRING will not work if being sent through FB_SQLDatabaseEvt, even if the syntax is correct.
• I_TcMessage will store any messages sent back from the database server. This will be helpful for identifying errors if there are any.

In the code execution section of MAIN, add the following basic state machine:

//Generate the query string.
//Change TableName to your table's name.
//Change column1 and column2 to the column names (include any other columns your table has, separated by ,).
//Change $'value1$' and $'value2$' to the values you wish to send.
//Note: $ is an escape character for a single quote. Single quotes indicate a string.

sQuery := 'INSERT INTO TableName (column1, column2) values ($'value1$', $'value2$')';

CASE nStep OF
    0:
        //Connect to the database at database ID 1
        IF fbDatabase.Connect(1) THEN
            nNextStep := 10;
        END_IF
    10:
        //Move the database message result to tcMessage
        tcMessage := fbDatabase.ipTcResult;
        //Go to disconnect step if there's an error
        IF fbDatabase.bError THEN
            nNextStep := 99;
        END_IF
        //Create a SQL command
        IF fbDatabase.CreateCmd(ADR(fbSqlCommand)) THEN
            nNextStep := 20;
        END_IF
    20:
        //Execute the SQL command of sQuery
        IF fbSQLCommand.Execute(ADR(sQuery), SIZEOF(sQuery)) >= 0 THEN
            tcMessage := fbSQLCommand.ipTcResult;
            nNextStep := 99;
        END_IF
    99:
        //Disconnect from the database
        IF fbDatabase.Disconnect() THEN
            nNextStep := 0;
        END_IF
END_CASE
 
//Advance the state machine step
nStep := nNextStep;

//Generate the query string.
//Change TableName to your table's name.
//Change column1 and column2 to the column names (include any other columns your table has, separated by ,).
//Change $'value1$' and $'value2$' to the values you wish to send.
//Note: $ is an escape character for a single quote. Single quotes indicate a string.

sQuery := 'INSERT INTO TableName (column1, column2) values ($'value1$', $'value2$')';

CASE nStep OF
    0:
        //Connect to the database at database ID 1
        IF fbDatabase.Connect(1) THEN
            nNextStep := 10;
        END_IF
    10:
        //Move the database message result to tcMessage
        tcMessage := fbDatabase.ipTcResult;
        //Go to disconnect step if there's an error
        IF fbDatabase.bError THEN
            nNextStep := 99;
        END_IF
        //Create a SQL command
        IF fbDatabase.CreateCmd(ADR(fbSqlCommand)) THEN
            nNextStep := 20;
        END_IF
    20:
        //Execute the SQL command of sQuery
        IF fbSQLCommand.Execute(ADR(sQuery), SIZEOF(sQuery)) >= 0 THEN
            tcMessage := fbSQLCommand.ipTcResult;
            nNextStep := 99;
        END_IF
    99:
        //Disconnect from the database
        IF fbDatabase.Disconnect() THEN
            nNextStep := 0;
        END_IF
END_CASE
 
//Advance the state machine step
nStep := nNextStep;

The state machine will do the following:

• Connect to the database with database ID 1. If you have multiple databases, you can change the database ID input to point to a specific database. You can find this ID by going back to the database configuration page (where you typed in the IP address, port number, etc.) and checking what the DBID is in the top left corner.
• Create the SQL command.
• Execute the SQL command according to sQuery
• Disconnect from the database

You can activate the configuration for the PLC project to start sending queries to the database!

4. Testing the Database Connection

4.1. Open TF6420 Configurator

After activating the configuration, the state machine should be looping through and inserting data into our table. We can check this by opening up the TF6420 Configurator application.

The database server and configuration should already be added from our steps to install the database server. If not, refer to the Adding the Database Project to a TwinCAT 3 Solution section.

4.2. Execute a SQL Select Query

Open the SQL Query Editor.

You should see a list of the SQL databases available from the DB configuration pool. You can click the refresh button if there are no databases currently connected. If that doesn’t work, make sure you’ve gone through the above steps and the configuration is activated.

Drop down the database to see all the available tables. You can right click on the table you’re writing to and click Select.

You can see that TF6420 has auto-generated a select statement for you. If you press the FB_SQLCommandEvt.ExecuteDataReturn button, you should see datasets populate in the results field to the right.

If these are the values you’re sending to the database then the database server is communicating properly!

For more information on some of the topics covered in this blog, please see the following:

• TF6420 Database Server Infosys Page
• SQL Expert Mode Function Block Infosys Page
• PostgreSQL Documentation
• Getting Started with a Beckhoff PLC

For information that was not discussed directly in this blog but could be good next steps, please see the following:

• MS SQL Stored Procedures Directly from Beckhoff TwinCAT PLC
• Getting Started with OPC UA in Beckhoff TwinCAT 3
• SQL Server Performance Troubleshooting
• NoSQL Expert mode Function Block Infosys Page

The post Getting Started with Database Server in Beckhoff TwinCAT 3 appeared first on DMC, Inc..