How large should be mysql innodb_buffer_pool_size?

Question

I have a busy database with solely InnoDB tables which is about 5GB in size. The database runs on a Debian server using SSD disks and I've set max connections = 800 which sometimes saturate and grind the server to halt. The average query per second is about 2.5K. So I need to optimize memory usage to make room for maximum possible connections.

I've seen suggestions that innodb_buffer_pool_size should be up to %80 of the total memory. On the other hand I get this warning from tuning-primer script:

Max Memory Ever Allocated : 91.97 G
Configured Max Per-thread Buffers : 72.02 G
Configured Max Global Buffers : 19.86 G
Configured Max Memory Limit : 91.88 G
Physical Memory : 94.58 G

Here are my current innodb variables:

| innodb_adaptive_flushing                          | ON                                                                                                                     |
| innodb_adaptive_hash_index                        | ON                                                                                                                     |
| innodb_additional_mem_pool_size                   | 20971520                                                                                                               |
| innodb_autoextend_increment                       | 8                                                                                                                      |
| innodb_autoinc_lock_mode                          | 1                                                                                                                      |
| innodb_buffer_pool_instances                      | 1                                                                                                                      |
| innodb_buffer_pool_size                           | 20971520000                                                                                                            |
| innodb_change_buffering                           | all                                                                                                                    |
| innodb_checksums                                  | ON                                                                                                                     |
| innodb_commit_concurrency                         | 0                                                                                                                      |
| innodb_concurrency_tickets                        | 500                                                                                                                    |
| innodb_data_file_path                             | ibdata1:10M:autoextend                                                                                                 |
| innodb_data_home_dir                              |                                                                                                                        |
| innodb_doublewrite                                | ON                                                                                                                     |
| innodb_fast_shutdown                              | 1                                                                                                                      |
| innodb_file_format                                | Antelope                                                                                                               |
| innodb_file_format_check                          | ON                                                                                                                     |
| innodb_file_format_max                            | Antelope                                                                                                               |
| innodb_file_per_table                             | ON                                                                                                                     |
| innodb_flush_log_at_trx_commit                    | 2                                                                                                                      |
| innodb_flush_method                               | O_DIRECT                                                                                                               |
| innodb_force_load_corrupted                       | OFF                                                                                                                    |
| innodb_force_recovery                             | 0                                                                                                                      |
| innodb_io_capacity                                | 200                                                                                                                    |
| innodb_large_prefix                               | OFF                                                                                                                    |
| innodb_lock_wait_timeout                          | 50                                                                                                                     |
| innodb_locks_unsafe_for_binlog                    | OFF                                                                                                                    |
| innodb_log_buffer_size                            | 4194304                                                                                                                |
| innodb_log_file_size                              | 524288000                                                                                                              |
| innodb_log_files_in_group                         | 2                                                                                                                      |
| innodb_log_group_home_dir                         | ./                                                                                                                     |
| innodb_max_dirty_pages_pct                        | 75                                                                                                                     |
| innodb_max_purge_lag                              | 0                                                                                                                      |
| innodb_mirrored_log_groups                        | 1                                                                                                                      |
| innodb_old_blocks_pct                             | 37                                                                                                                     |
| innodb_old_blocks_time                            | 0                                                                                                                      |
| innodb_open_files                                 | 300                                                                                                                    |
| innodb_purge_batch_size                           | 20                                                                                                                     |
| innodb_purge_threads                              | 0                                                                                                                      |
| innodb_random_read_ahead                          | OFF                                                                                                                    |
| innodb_read_ahead_threshold                       | 56                                                                                                                     |
| innodb_read_io_threads                            | 4                                                                                                                      |
| innodb_replication_delay                          | 0                                                                                                                      |
| innodb_rollback_on_timeout                        | OFF                                                                                                                    |
| innodb_rollback_segments                          | 128                                                                                                                    |
| innodb_spin_wait_delay                            | 6                                                                                                                      |
| innodb_stats_method                               | nulls_equal                                                                                                            |
| innodb_stats_on_metadata                          | ON                                                                                                                     |
| innodb_stats_sample_pages                         | 8                                                                                                                      |
| innodb_strict_mode                                | OFF                                                                                                                    |
| innodb_support_xa                                 | ON                                                                                                                     |
| innodb_sync_spin_loops                            | 30                                                                                                                     |
| innodb_table_locks                                | ON                                                                                                                     |
| innodb_thread_concurrency                         | 4                                                                                                                      |
| innodb_thread_sleep_delay                         | 10000                                                                                                                  |
| innodb_use_native_aio                             | ON                                                                                                                     |
| innodb_use_sys_malloc                             | ON                                                                                                                     |
| innodb_version                                    | 1.1.8                                                                                                                  |
| innodb_write_io_threads                           | 4                                                                                                                      |

A side note that might be relevant: I see that when I try to insert a large post (say over 10KB) from Drupal (which sits on a separate web server) to database, it lasts forever and the page does not return correctly.

Regarding these, I'm wondering what should be my innodb_buffer_pool_size for optimal performance. I appreciate your suggestions to set this and other parameters optimally for this scenario.

score 363 · Accepted Answer · edited Feb 10 '20 at 17:03

Your innodb_buffer_pool_size is enormous. You have it set at 20971520000. That's 19.5135 GB. If you only have 5GB of InnoDB data and indexes, then you should only have about 8GB. Even this may be too high.

Here is what you should do. First run this query

SELECT CEILING(Total_InnoDB_Bytes*1.6/POWER(1024,3)) RIBPS FROM
(SELECT SUM(data_length+index_length) Total_InnoDB_Bytes
FROM information_schema.tables WHERE engine='InnoDB') A;

This will give you the RIBPS, Recommended InnoDB Buffer Pool Size, based on all InnoDB Data and Indexes, with an additional 60%.

For Example

mysql>     SELECT CEILING(Total_InnoDB_Bytes*1.6/POWER(1024,3)) RIBPS FROM
    ->     (SELECT SUM(data_length+index_length) Total_InnoDB_Bytes
    ->     FROM information_schema.tables WHERE engine='InnoDB') A;
+-------+
| RIBPS |
+-------+
|     8 |
+-------+
1 row in set (4.31 sec)

mysql>

With this output, you would set the following in /etc/my.cnf

[mysqld]
innodb_buffer_pool_size=8G

Next, service mysql restart

After the restart, run MySQL for a week or two. Then, run this query:

SELECT (PagesData*PageSize)/POWER(1024,3) DataGB FROM
(SELECT variable_value PagesData
FROM information_schema.global_status
WHERE variable_name='Innodb_buffer_pool_pages_data') A,
(SELECT variable_value PageSize
FROM information_schema.global_status
WHERE variable_name='Innodb_page_size') B;

This will give you how many actual GB of memory is in use by InnoDB Data in the InnoDB Buffer Pool at this moment.

I have written about this before : What to set innodb_buffer_pool and why..?

You could just run this DataGB query right now rather than reconfiguring, restarting and waiting a week.

This value DataGB more closely resembles how big the InnoDB Buffer Pool should be + (percentage specified in innodb_change_buffer_max_size). I am sure this will be far less than the 20000M you have reserved right now. The savings in RAM can be used for tuning other things like

CAVEAT #1

This is very important to note: At times, InnoDB may require an additional 10% over the value for the innodb_buffer_pool_size. Here is what the MySQL Documentation says on this:

The larger you set this value, the less disk I/O is needed to access data in tables. On a dedicated database server, you may set this to up to 80% of the machine physical memory size. Be prepared to scale back this value if these other issues occur:

Competition for physical memory might cause paging in the operating system.

InnoDB reserves additional memory for buffers and control structures, so that the total allocated space is approximately 10% greater than the specified size.

The address space must be contiguous, which can be an issue on Windows systems with DLLs that load at specific addresses.

The time to initialize the buffer pool is roughly proportional to its size. On large installations, this initialization time may be significant. For example, on a modern Linux x86_64 server, initialization of a 10GB buffer pool takes approximately 6 seconds. See Section 8.9.1, “The InnoDB Buffer Pool”.

CAVEAT #2

I See the following values in your my.cnf

| innodb_io_capacity                                | 200 |
| innodb_read_io_threads                            | 4   |
| innodb_thread_concurrency                         | 4   |
| innodb_write_io_threads                           | 4   |

These number will impede InnoDB from accessing multiple cores

Please set the following:

[mysqld]
innodb_io_capacity = 2000
innodb_read_io_threads = 64
innodb_thread_concurrency = 0
innodb_write_io_threads = 64

I have written about this before in the DBA StackExchange

May 26, 2011: About single threaded versus multithreaded databases performance
Sep 12, 2011: Possible to make MySQL use more than one core?
Sep 20, 2011: Multi cores and MySQL Performance

I just answered a question like this in Server Fault using a more concise formula:

SELECT CONCAT(CEILING(RIBPS/POWER(1024,pw)),SUBSTR(' KMGT',pw+1,1))
Recommended_InnoDB_Buffer_Pool_Size FROM
(
    SELECT RIBPS,FLOOR(LOG(RIBPS)/LOG(1024)) pw
    FROM
    (
        SELECT SUM(data_length+index_length)*1.1*growth RIBPS
        FROM information_schema.tables AAA,
        (SELECT 1.25 growth) BBB
        WHERE ENGINE='InnoDB'
    ) AA
) A;

Thanks for this great post! Your formula starting with `SELECT (PagesData*PageSize)/POWER(1024,3) DataGB FROM...` generates the following error on MySQL 5.7: "*The 'INFORMATION_SCHEMA.GLOBAL_STATUS' feature is disabled; see the documentation for 'show_compatibility_56'*". Would you have an updated version by any chance? — BenMorel, Jun 27 '16 at 11:31
I get 307 RIBPS and 264G. That's mean that I need 307GB of RAM? — E_Blue, Sep 06 '16 at 20:50
More like 264G. But you should have enough RAM for that, else give the mentioned 80% of your RAM to mysql, depending on what else runs on the system. — sjas, Sep 28 '16 at 08:23
@Benjamin: As of MySQL 5.7.6 the information_schema is merged into performance_schema. So just change "information_schema" to "performance_schema" in the query to make it work. Source: https://dev.mysql.com/doc/refman/5.7/en/status-table.html — Ralph Bolton, Mar 13 '17 at 10:59
I've run this in a bunch of my client's servers. The recommendations are a bit on the high side in a lot of cases - on some servers, after months of uptime the current pool is only 60% utilised, so (probably) doesn't really need expanding (evidently they don't get used that much). Getting this stuff onto graphs will give better information, but for an instantaneous 'rule of thumb' this does really well. As with all per. tuning: Caveat DBA ;-) — Ralph Bolton, Mar 13 '17 at 11:46
@RolandoMySQLDBA, After reading the docs, as best i can tell the `innodb_read_io_threads` and `innodb_write_io_threads` setting will make no difference when `innodb_use_native_aio = ON` as in this case. From https://dev.mysql.com/doc/refman/5.7/en/innodb-linux-native-aio.html - _With native AIO, query threads dispatch I/O requests directly to the operating system, thereby removing the limit imposed by the number of background threads._ Any thoughts on that? — Will, Oct 09 '17 at 14:45
what is your meaning about `InnoDB data and indexes`. data will collected at the time,this week i have about 10 gig in my database and next week my data is 20 gig, you mean in must change this value every month !? — peiman F., Jun 08 '19 at 21:15
Caveat: "enormous" is relative. That word was written in 2012, many years ago. — Rick James, Jan 26 '20 at 20:26
@RickJames you are right. Today you have the Amazon RDS Instance db.x1e.32xlarge with 3.9TB of RAM with a buffer pool 2.85 TB (based on default of 3/4 of RAM). Now, that's enourmous, — RolandoMySQLDBA, Jan 27 '20 at 02:10
@RolandoMySQLDBA - and in 8 more years, 2.85 TB will seem tiny. That's life. — Rick James, Jan 27 '20 at 06:06
That `SELECT` may give a desired size for the buffer_pool, but _it must be capped by some percentage of RAM_. — Rick James, Oct 29 '20 at 21:44

score 25 · Answer 2 · edited Jan 28 '21 at 13:01

25

Something like this? Using SHOW VARIABLES and SHOW GLOBAL STATUS:

Expression: innodb_buffer_pool_size / _ram
Meaning: % of RAM used for InnoDB buffer_pool
Recommended range: 60~80%

Expression: Innodb_buffer_pool_reads / Innodb_buffer_pool_read_requests
Meaning: Read requests that had to hit disk
Recommended range: 0-2%
What to do if out of range: Increase innodb_buffer_pool_size if you have enough RAM.

Expression: Innodb_pages_read / Innodb_buffer_pool_read_requests
Meaning: Read requests that had to hit disk
Recommended range: 0-2%
What to do if out of range: Increase innodb_buffer_pool_size if you have enough RAM.

Expression: Innodb_pages_written / Innodb_buffer_pool_write_requests
Meaning: Write requests that had to hit disk
Recommended range: 0-15%
What to do if out of range: Check innodb_buffer_pool_size

Expression: Innodb_buffer_pool_reads / Uptime
Meaning: Reads
Recommended range: 0-100/sec.
What to do if out of range: Increase innodb_buffer_pool_size?

Expression: (Innodb_buffer_pool_reads + Innodb_buffer_pool_pages_flushed) / Uptime
Meaning: InnoDB I/O
Recommended range: 0-100/sec.
What to do if out of range: Increase innodb_buffer_pool_size?

Expression: Innodb_buffer_pool_pages_flushed / Uptime
Meaning: Writes (flushes)
Recommended range: 0-100/sec.
What to do if out of range: Increase innodb_buffer_pool_size?

Expression: Innodb_buffer_pool_wait_free / Uptime
Meaning: Counter for when there are no free pages in buffer_pool. That is, all pages are dirty.
Recommended range: 0-1/sec.
What to do if out of range: First be sure innodb_buffer_pool_size is set reasonably; if still trouble, decrease innodb_max_dirty_pages_pct

edited Jan 28 '21 at 13:01

Hosam Aly

105
4

answered Feb 07 '15 at 01:02

Rick James

73,608
4
41
101

Thanks @Rick for the nice comment. What `innodb_buffer_pool_size` value specifies? Actual size or configured one? – joker Dec 21 '17 at 12:01
1

@joker - `innodb_buffer_pool_size` indicates the max size. In a typical server, the "buffer pool" starts small, but quickly grows to that max size and stays there. Note: If that is bigger than RAM (or even close), then that leads to swapping, which is terrible for performance. – Rick James Apr 20 '19 at 18:47
So you're saying that innodb_buffer_pool_size is important? :) Nice helpful answer – mikato Mar 12 '20 at 17:07
@mikato - It is the main thing to tune for performance. Too small --> more I/O. Bigger than RAM --> swapping or OOM crash. – Rick James Mar 13 '20 at 00:21
Thanks! Is there a way to calculate these expressions in MySQL 5.x directly, without extracting them one at a time with `show global status like ...`? – Hosam Aly Jan 26 '21 at 19:06
1

@HosamAly - Not really. I do `SHOW GLOBAL STATUS;` into PHP code, then convert the results to an associative array. (Note: capitalized values come from `STATUS`; lowercase come from `VARIABLES`; `_ram` is not available from either.) – Rick James Jan 26 '21 at 19:12
@HosamAly - and... I currently have over 200 expressions, and they are being added to / changed frequently as I discover new things. To handle that many, I actually use some regexps and `eval()`. – Rick James Jan 26 '21 at 19:14
1

@HosamAly - and... The `SHOW GLOBAL STATUS` records counts, etc from when MySQL was last restarted. There are situations where it would be better to grab the status at specific times (like before and after a spike in traffic), then subtract, and analyze. Even without that sophistication, I do find useful info. – Rick James Jan 26 '21 at 19:17
1

@HosamAly - and... Be aware that SSD runs a lot faster than HDD; the recommendations listed here are 6 years old, and mostly aimed at HDD. (Rule of Thumb: SSD is 10 times as fast -- _but_ there is a lot of variation.) – Rick James Jan 26 '21 at 19:19
@HosamAly - Also, what was the rationale about deleting the entry for innodb_pages_read? – Rick James Jan 26 '21 at 19:29
Thanks a lot for your response, @RickJames. And I apologise sincerely for deleting the entry for `innodb_pages_read`. My eyes hadn't seen the difference between that expression and the one before it because they shared the same meaning, recommended range, and resolution. I've now reverted it back. Please accept my apologies. – Hosam Aly Jan 28 '21 at 00:21
@HosamAly - No problem; apology accepted. I was willing to listen to an argument against my entry; I do make mistakes and/or learn from other people's experiences. – Rick James Jan 28 '21 at 00:25
Following this advice, I sped up my MySQL processing by more than 100%. Thanks! – Allen Gingrich Jun 27 '22 at 19:45
@AllenGingrich - Great! I've added a lot of tests in the 7 years since this Answer. Provide `SHOW GLOBAL STATUS` and `SHOW VARIABLES` and RAM size; I'll review your setup further. – Rick James Jun 27 '22 at 21:06
Is there a variant, that selects all variables in one query? – M-A-X Feb 26 '23 at 19:32
@M-A-X - `SELECT * FROM performance_schema.global_variables;` in 8.0. See also `global_status`. Before that, similar tables existed in `infomration_schema`. Such as `SELECT * FROM performance_schema.global_status WHERE VARIABLE_NAME LIKE "Innodb_buffer_pool%";` – Rick James Feb 26 '23 at 20:22
And, with some self-joins, you could evaluate those expressions. – Rick James Feb 26 '23 at 20:25

score 7 · Answer 3 · answered Oct 22 '12 at 23:19

Your title asks about innodb_buffer_pool_size, but I suspect that is not the real problem. (Rolando commented on why you have set it big enough, even too big.)

I've set max connections = 800 which sometimes saturate and grind the server to halt.

That is unclear. 800 users in "Sleep" mode has virtually zero impact on the system. 800 active threads would be a disaster. How many threads are "running"?

Are the threads blocking each other? See SHOW ENGINE INNODB STATUS for some clues on deadlocks, etc.

Are any queries showing up in the slowlog? Let's optimize them.

What version are you using? XtraDB (a drop-in replacement for InnoDB) does a better job of using multiple cores. 5.6.7 does an even better job.

innodb_buffer_pool_instances -- change this to 8 (assuming a 20G buffer_pool); it will cut back slightly on the Mutex contention.

Are you I/O bound or are you CPU bound? The solutions are radically different, depending on your answer.

SSD -- It might be better if all the log files were on non-SSD drives.

confused: so should we follow the mysql recommended size 70% or the one that rolando recommended above by using the performance schema and then getting the actual used size? — Muhammad Omer Aslam, Jan 19 '22 at 14:06
@MuhammadOmerAslam - Either way. There is no "perfect" size. I say 70% because that works safely for any machine but the smallest. Measuring the actual dataset size also works nicely unless that leads to using too much RAM (and causing swapping). So... The smaller of the computation or 70%. — Rick James, Jan 19 '22 at 16:50

score 7 · Answer 4 · edited Jun 03 '19 at 10:05

More memory is always better, but in my experience most of the times buffer pool size should not fit your data size. Many tables are inactive most of the times, like backup tables lying around, so innodb buffer pool size should rather fit you acive data size.

The time frame you specify for active pages influences the performance, but there's an optimal point, where you won't get that more performance for a bigger buffer size. You could estimate/calculate/measure that by show engine innodb status

score 2 · Answer 5 · answered Feb 26 '23 at 20:15

@RickJames answer in sql:

SELECT
    'Innodb_buffer_pool_reads / Innodb_buffer_pool_read_requests' AS Expression,
    'Read requests that had to hit disk' AS Meaning,
    ROUND((SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_buffer_pool_reads'
    )
    /
    (SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_buffer_pool_read_requests'
    ) * 100, 2) AS Val,
    '0-2%' AS Recommend,
    'Increase innodb_buffer_pool_size if you have enough RAM.' AS WhatToDoIfOutOfRange
UNION
SELECT
    'Innodb_pages_read / Innodb_buffer_pool_read_requests' AS Expression,
    'Read requests that had to hit disk' AS Meaning,
    ROUND((SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_pages_read'
    )
    /
    (SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_buffer_pool_read_requests'
    ) * 100, 2) AS Val,
    '0-2%' AS Recommend,
    'Increase innodb_buffer_pool_size if you have enough RAM.' AS WhatToDoIfOutOfRange
UNION
SELECT
    'Innodb_pages_written / Innodb_buffer_pool_write_requests' AS Expression,
    'Write requests that had to hit disk' AS Meaning,
    ROUND((SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_pages_written'
    )
    /
    (SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_buffer_pool_write_requests'
    ) * 100, 2) AS Val,
    '0-15%' AS Recommend,
    'Increase innodb_buffer_pool_size if you have enough RAM.' AS WhatToDoIfOutOfRange
UNION
SELECT
    'Innodb_buffer_pool_reads / Uptime' AS Expression,
    'Reads' AS Meaning,
    ROUND((SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_buffer_pool_reads'
    )
    /
    (SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Uptime'
    )) AS Val,
    '0-100/sec' AS Recommend,
    'Increase innodb_buffer_pool_size if you have enough RAM.' AS WhatToDoIfOutOfRange
UNION
SELECT
    '(Innodb_buffer_pool_reads + Innodb_buffer_pool_pages_flushed)  / Uptime' AS Expression,
    'InnoDB I/O' AS Meaning,
    ROUND(((SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_buffer_pool_reads'
    )
    +
    (SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_buffer_pool_pages_flushed'
    ))
    /
    (SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Uptime'
    )) AS Val,
    '0-100/sec' AS Recommend,
    'Increase innodb_buffer_pool_size if you have enough RAM.' AS WhatToDoIfOutOfRange
UNION
SELECT
    'Innodb_buffer_pool_pages_flushed / Uptime' AS Expression,
    'Writes (flushes)' AS Meaning,
    ROUND((SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_buffer_pool_pages_flushed'
    )
    /
    (SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Uptime'
    )) AS Val,
    '0-100/sec' AS Recommend,
    'Increase innodb_buffer_pool_size if you have enough RAM.' AS WhatToDoIfOutOfRange
UNION
SELECT
    'Innodb_buffer_pool_wait_free / Uptime' AS Expression,
    'Counter for when there are no free pages in buffer_pool. That is, all pages are dirty.' AS Meaning,
    ROUND((SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Innodb_buffer_pool_wait_free'
    )
    /
    (SELECT
        VARIABLE_VALUE
    FROM
        information_schema.GLOBAL_STATUS
    WHERE
        VARIABLE_NAME = 'Uptime'
    )) AS Val,
    '0-1/sec' AS Recommend,
    'First be sure innodb_buffer_pool_size is set reasonably; if still trouble, decrease innodb_max_dirty_pages_pct' AS WhatToDoIfOutOfRange

How large should be mysql innodb_buffer_pool_size?

5 Answers5

CAVEAT #1

CAVEAT #2

Linked

Related