8 Replies Latest reply on Nov 10, 2019 9:58 AM by Lisa Keeler

    Appliance crashed due to large size/amount of db transaction logs

    Jacek Szlaczka
      Share This:

      Hello,

       

      Our production appliance has crashed due to lack of space in /usr/ directory.
      After looking into it, tideway.db/logs/ directory takes over 30GB of space.

       

      I managed to free up 1.2GB of space, but the services still won't start due to lack of disk space.

       

      We've raised a support ticket, but they recommended to run tw_ds_offline_compact. We did that but as you may guess, it did not reduce the size of transaction logs.

       

      Can we just move these logs from /usr/tideway/var/localdisk/tideway.db/logs/ to a different location, start services and then move them back to the original location and let the appliance process them?

       

      Do we have other options? We could also create a symlink and put these logs in the datastore partition I guess?

        • 1. Re: Appliance crashed due to large size/amount of db transaction logs
          Jacek Szlaczka

          We've moved the transaction logs to a different location and created a symlink to it, however after starting services the size of transaction logs has grown from 30GB to 60GB in just 30 minutes!

           

          How can we check what causes this growth? If the trend continues, it will shut down our environment with no easy way to recover this time...

          • 2. Re: Appliance crashed due to large size/amount of db transaction logs
            Lisa Keeler

            I think that your Customer support representative did not understand the problem.

             

            You should reopen the ticket and request internal KA#000149563.

             

            The KA tells you that the recommended approach is to add a new disk, and then use tw_disk_utils to move the transaction logs to the new disk.

             

            How to use tw_disk_utils is described in KA:  000082640

            Discovery: How to use tw_disk_utils; Cannot access UI, Insufficient disk space to start services. The datastore files or datastore transaction logs are too large.

             

            The alternate approach (not the recommended approach) is to move the location of the datastore transaction logs to an existing location where you have more space.

            The internal KA tells you how to do that.

            If you are taking that approach, then you should ask for help (with webex) on your support ticket.

             

            If you take either approach, I don't recommend that you move your transaction logs back into the /usr partition after the crisis is over...  Because it will likely fill up the /usr partition again at a later time.

             

            And, you seem to know already NOT to delete any transaction logs.  (Good job on that)

             

             

             

            1 of 1 people found this helpful
            • 3. Re: Appliance crashed due to large size/amount of db transaction logs
              Lisa Keeler

              Jared,

               

              It probably will stop growing at some point. 

               

              But, things to review in the support ticket:

              1) what is your version of Discovery

              2) what is your TKU version

              3) find the data that is causing the file to be so large

                    do that by taking the filename of the large file, and copy the part before the underscore.  Then, paste that into the search box in Discovery.

                    What type of node does it find? 

              1 of 1 people found this helpful
              • 4. Re: Appliance crashed due to large size/amount of db transaction logs
                Jacek Szlaczka

                Well for now we are seeing 1GB per minute growth of transaction logs folder on cluster coordinator. There's 500GB of free disk space in the partition, so we won't survive the weekend if it continues.

                However, the 2nd node of cluster has no issues at all, very fishy if you ask me.

                 

                Hopefully the customer support can help...

                • 5. Re: Appliance crashed due to large size/amount of db transaction logs
                  Jacek Szlaczka

                  BMC Discovery 11.3,

                  TKU:TKU-2019-08-1-ADDM-11.3+

                   

                  There's no underscores in file names:

                   

                  [tideway@server] du -a /mnt/addm/db_data/data/datadir/logs

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000309

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000271

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000121

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000264

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000149

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000284

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000319

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000143

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000420

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000429

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000298

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000289

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000151

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000366

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000229

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000202

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000159

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000247

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000163

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000403

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000394

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000158

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000188

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000358

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000448

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000241

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000305

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000426

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000391

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000327

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000411

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000276

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000184

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000416

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000234

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000376

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000318

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000359

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000444

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000155

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000311

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000397

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000180

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000387

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000135

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000449

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000137

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000392

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000147

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000373

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000342

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000274

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000123

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000333

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000404

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000223

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000331

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000398

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000145

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000128

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000301

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000425

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000415

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000281

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000279

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000270

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000249

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000194

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000167

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000131

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000432

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000162

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000287

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000402

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000431

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000153

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000372

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000239

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000324

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000374

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000129

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000414

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000262

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000346

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000144

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000175

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000340

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000268

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000361

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000208

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000337

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000154

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000296

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000434

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000275

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000248

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000390

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000439

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000253

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000353

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000310

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000401

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000389

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000255

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000282

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000205

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000299

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000430

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000326

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000378

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000169

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000322

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000170

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000189

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000436

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000232

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000197

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000179

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000178

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000132

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000303

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000345

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000204

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000171

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000335

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000312

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000220

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000265

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000307

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000300

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000173

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000351

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000352

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000313

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000350

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000150

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000196

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000286

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000285

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000330

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000273

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000213

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000348

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000183

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000133

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000209

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000225

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000242

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000200

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000141

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000291

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000417

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000240

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000190

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000148

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000224

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000238

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000341

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000152

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000267

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000227

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000201

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000138

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000435

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000146

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000186

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000445

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000231

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000252

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000399

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000181

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000261

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000433

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000124

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000222

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000256

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000168

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000243

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000424

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000130

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000245

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000323

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000440

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000306

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000297

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000418

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000338

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000280

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000206

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000356

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000304

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000382

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000294

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000370

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000257

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000230

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000363

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000217

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000140

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000277

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000375

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000160

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000218

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000410

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000134

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000377

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000212

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000413

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000174

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000317

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000406

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000156

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000258

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000182

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000302

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000320

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000136

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000260

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000325

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000393

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000166

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000195

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000321

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000384

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000367

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000334

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000388

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000339

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000408

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000278

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000385

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000193

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000185

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000380

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000349

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000198

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000215

                  58372   /mnt/addm/db_data/data/datadir/logs/log.0000000451

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000395

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000423

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000357

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000191

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000332

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000365

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000165

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000157

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000295

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000369

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000187

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000235

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000254

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000290

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000329

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000177

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000127

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000316

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000407

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000364

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000447

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000221

                  262124  /mnt/addm/db_data/data/datadir/logs/log.0000000126

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000308

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000442

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000409

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000362

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000443

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000427

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000441

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000344

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000192

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000419

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000328

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000246

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000244

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000412

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000386

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000203

                  262128  /mnt/addm/db_data/data/datadir/logs/log.0000000122

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000315

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000214

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000251

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000428

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000368

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000172

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000237

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000354

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000226

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000400

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000236

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000139

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000269

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000176

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000161

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000383

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000343

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000288

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000199

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000250

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000219

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000336

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000405

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000283

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000211

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000437

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000216

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000292

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000272

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000438

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000371

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000293

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000360

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000381

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000314

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000421

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000422

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000266

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000347

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000355

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000207

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000142

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000446

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000259

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000210

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000125

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000263

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000450

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000164

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000233

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000379

                  262148  /mnt/addm/db_data/data/datadir/logs/log.0000000228

                  262144  /mnt/addm/db_data/data/datadir/logs/log.0000000396

                  86566248        /mnt/addm/db_data/data/datadir/logs

                  • 6. Re: Appliance crashed due to large size/amount of db transaction logs
                    Lisa Keeler

                    Of course you are right.  No underscores.

                     

                    Did you recently change the setting for how long to keep History?

                    In model maintenance page.

                    • 7. Re: Appliance crashed due to large size/amount of db transaction logs
                      Jacek Szlaczka

                      Not so recently, it was changed around 2 months ago from 28 days to 14 days.

                       

                      I think the situation got stable. Transaction logs got to 100GB and then down to 10GB which finally made me more calm about the whole thing.

                      I'll monitor it during the weekend.

                      • 8. Re: Appliance crashed due to large size/amount of db transaction logs
                        Lisa Keeler

                        1) RE: Moving the transaction logs :

                        You said:> We've moved the transaction logs to a different location and created a symlink to it, however after starting services the size of transaction logs has grown from 30GB to 60GB in just 30 minutes!

                         

                        LISA> When you move the location, you also need to edit the /usr/tideway/etc/link.conf file per (internal) KA: 000149563

                         

                         

                        2) RE: The cause of the problem:

                        28 and 14 days are the "DDD Removal" settings, shown in red.

                         

                        I was asking about the setting in green below:  "Time before history entries are purged"

                        (Did you change it from the default of "Never" to something else?)

                         

                         

                        You could look at the Audit history to see if the setting was changed.

                         

                        We have a known problem that sometimes the transaction logs fill up the disk when the History setting is lowered.

                         

                        We have an internal KA about that problem:  000164304

                         

                        Problem

                        PROBLEM:  The partition which contains the datastore transaction logs fills up.
                        The datastore transaction logs are located here (at a logical link):  /usr/tideway/var/tideway.db/log
                        OOTB, the partition which holds the datastore transaction logs is the /usr partition.  But, the location can be moved either from the Disk Configuration UI page, tw_disk_utils, or manually:  See KA#000149563.

                         

                        CAUSE:  After the "Time before history entries are purged" setting is lowered, the disk can fill up with datastore transaction logs.
                        In Administration > Model Maintenance the default value for "Time before history entries are purged" is "Never".
                        (The UI setting corresponds to the DATASTORE_HISTORY_PURGE_DAYS setting in tw_options.)

                         

                        After customer lowers the setting, the large amount of history nodes being purged can result in a spike of datastore transaction logs.  This can fill up the partition holding this data.

                         

                        This problem is most often observed while purging the history of the Pattern nodes.

                         

                        NOTE:  After the initial spike in purging of history entries, the problem will subside.  This is a one-time event after the "Time before history entries are purged" setting is lowered.

                        Solution

                         

                        To resolve the immediate crisis, customer must move his transaction log location to a larger disk.  (See KA#000149563)

                         

                        To help prevent the problem in the future, customer can lower the "Checkpoint Interval" setting from the default of 15 minutes.  (The setting is at the bottom of the Model Maintenance page).

                         

                        Lowering the "Checkpoint Interval" causes the model to process the transaction logs more frequently.  After a transaction log is processed, the log is removed from the disk.
                        It is less likely that the disk will become full with more frequent checkpoints.

                         

                        NOTE:  It is not recommended for Customer to set the "Time before history entries are purged" back to "Never".
                        There was a reason for changing the setting.  The reason is so that the datastore will be smaller.
                        Restoring the setting to "Never" will only lead to the problem again in the future after the setting is lowered again.
                        At the point of a full disk, customer has already "paid the price" for changing the setting.  It is recommended to move forward with the new setting.

                        1 of 1 people found this helpful