[ b / kemono / coomer ]

/kemono/ - kemono.party

Kemono Development and Discussion
Name
Email
Subject
Comment
Verification
File
Password (For file deletion.)

File: 1635804536483.jpg (73.73 KB, 550x367, server_spaghetti_1.jpg)

 No.8965

See here for words on Maintenance and works being performed on the site!

 No.15933

Message from Administration:
Kemono is now unblocked from Verizon, OpenDNS, and various global ISP bans. Additionally, the site should be a bit faster.

Also, If you are using any downloaders, you should no longer need cookies, so feel free to delete them if you'd like.

 No.16342

As you may have noticed, the importer has been disabled. This is due to work being carried out on the site. Expect it to be back up and running when work is concluded. You can start your import now, it will be added to the queue and be imported as soon as the importer is re-enabled.
Sorry for any inconvenience. Thanks for understanding.

 No.16365

Works have been finished and the importer re-enabled. Site should be a bit speedier now too, but please keep your downloading to a reasonable level. Thanks for your patience.

 No.16589

Due to strange behaviour on our data server Hdds, we are doing several checks to make sure it isn't anything we should worry about, to speed things up we are cutting all connections to the data server, cache servers are still operational and if your file was cached then it will be served without a problem, for non cached files you will notice 5XX Type HTTP errors.

We are estimating that the downtime will be around 1-2 days when the checks are running, during this time imports will be put into the queue and will get run after downtime is over, other site functions will continue to work.

 No.16630

Work has now condluded. For more information, see >>16624 as he has a good explanation and some pretty good suggestions for everyone.

 No.17618

So all data servers were just checked by me personally and those with badly behaving hdds got their drives swapped for new ones.

Enjoy the new kemono download speed!

 No.17640

30-60min maintenance work on the backend servers. Some new data might not load for the time being.

 No.17644

Maintenance is finished.

 No.17668

File: 1645664021113.gif (80.22 KB, 220x164, monkey-computer.gif)

The storage array is currently under maintenance. Imports might be halted for the time being.

 No.17688

File: 1645756070014.jpg (230.25 KB, 1907x1079, FEWlb3q.jpg)

Hoo boy, there is no end to these things. Here is a short list that will be followed up with a longer explanation.
Are things fucked? Yes. How fucked? It is fixable, but will take time. Is there data loss? Unlikely, need more time but we have backups.
Will imports work? Yes, I'll make it work in ~12h. How long will it take? Don't know, maybe a week or two. I'll make it so that you are able to get the data somehow.
End of the Month import will run. Just submit your key and once everything is running it will import.


Now the longer explanation while things run in the background.
While the periodic flushing of data from RAID1 to the disk array was ongoing, the disks decided to hard reset themselves to a point where the filesystems are now in a corrupted state.
The corruption seems to only apply to the data that was being flushed to the disks, which is still available on the RAID1 mount. Given the history of this happening before,
alas the filesystem was repairable at that time, this time it was much worse and I refuse to do any modifications to the filesystems/disks. As not to fuck it beyond the current state.
The metadata and data up to the point of the "periodic flushing" seems to be doing fine. The hashes match, the data is readable. This is also the reason why you can view all the data right now.
The disks behave when you read from them at full speed, but god is not on your side when you start writing to them. The CRC errors hint at the backplane shitting itself and leaving skidmarks on the disks.
There is also the possibility of the NCQ(+linux own queue) shitting itself and linux resetting the disk. In my days I experienced a multitude of disks resetting themselves so hard that you would need to physically disconnect and re-connect them, due to fucked queueing.
But in this case the "hard reset" did bring the disks back, so I can only guess that the FW got better or this is a linux thing that is causing problems. And because the drives were recently released, nothing is to be found on mailing lists.

Now, is the data safe? Yes it is. KP data is backed up periodically and the last backup was a bit before the "periodic flushing". The data on RAID1 is currently being backed up. So this is the least of my worries, I hope.

And finally about importing. I will have to juggle a lot, manually prime the cache servers with new data, backup and replicate the data to other servers and a lot more.
BUT I think I will be able to deliver all the data that is being imported until the issues are resolved. I can't promise a smooth ride, but I think I can bridge the issues we have right now.

fml, I just want a comfy ride without any issues, I'm tired. I just want some enterprise grade storage systems with support and not care about anything.
Also, I think I forgot to mention but this is only a data server. Nothing else is going on there. So the other parts are not affected by this.

 No.17787

File: 1646193703592.jpg (370.25 KB, 900x903, EyyBiHkWUAEq1Hp.jpg)

Slowly chugging along. The important data has been copied, delta is being applied. After that even more data integrity validation is needed.
My estimate is end of the week, likely earlier. But it's not like you will notice anything, maybe a small downtime. Well, whatever.

 No.17837

File: 1646422749555.jpg (112.23 KB, 444x460, 1634673234730.jpg)

You fix one thing and then another one pops up. Gotta love the timing.
Nonetheless ~55% of data integrity validation is done. Hey, I did mention that it will be slow.
In regards to the download problems of some files right now… your complaints fall on deaf ears.
I'm only making sure that imports will work. The rest will wait until I'm done with recovery.
And I'm still optimistic that I might be done by the end of the week. No promises.

 No.17885

File: 1646598956733.jpg (83.59 KB, 895x504, cg1.JPG)

All data has been verified and nothing is missing nor is anything corrupt. So one could say we are "done", but I'll postpone it by day.
I've noticed an anomaly that is not dangerous, but rather annoying. I want to figure out the cause of this, before going prod with this.

 No.17908

File: 1646675665336.png (1.11 MB, 822x1504, 1640159528140.png)

Almost done. All cache servers are now connected to the new system.
Going by the stats, the cache servers were starved quite bad.

Imports have been suspended for the next few hours to see how the system behaves under real load.

 No.17916

File: 1646695121259.png (1.74 MB, 1520x1119, That_Fucking_Bird.png)

Looks like things are now stable, but I will have to monitor the whole thing for a few more days to be sure.
In other news, imports are back online. Tomorrow, at some point in time, there will be an unannounced ~30min downtime.
I need to adjust the cache servers and from there, possibly ~24h of varying download speeds. Will also serve as a stress test for the data server.

 No.17938

File: 1646773858697.jpg (95.74 KB, 726x520, monster_girls.jpg)

Excluding the screaming status page and me not being able to ssh into the data server, I can say with confidence that we are going at MAX SPEED right now.
Well, it will ease up at some point for sure… a few hours should do it, I think. Either way, expect some slowdowns. In ~24-48h it should calm down completely.

 No.18054

File: 1647042560826.jpeg (333.65 KB, 708x1000, b4edc92a7a382eab3df4214ca….jpeg)

With the whopping 14 cache servers we have now (about 14 gbit of total uplink) the download speeds should be above the expectations you had with kemono, no more 429 and no more crazy slow download speeds.

I myself tested it out and it was going at a steady 10-20mb/s per download.

If you do experience any slowdowns be sure to tell us here >>14747 with a link to the content that was slow.

 No.18298

File: 1647723801749.jpg (130.55 KB, 1280x938, photo_2022-03-12_20-58-05.jpg)

Changed the filtering rules for the websites, gallery-dl should no longer trigger the anti ddos protection and work as expected.

 No.18654

File: 1648442921880.jpg (25.28 KB, 800x480, suggestion-box-improve-bus….jpg)

Have ideas for what could be next in regards to the importer? Suggest platforms that could be implemented into the importer in the following form.
This isn't a guarantee that they will be implemented. This is only a way to gauge interest for what MIGHT be implemented in the future.

Link: https://forms.gle/DQSfhJMG4AmZLmpAA

 No.18697

File: 1648497359366.webm (1.16 MB, 720x720, fd7bd8cf8accb503fdc045e43….webm)

This is a forwarded reply from >>18576

I don't have anything to directly add to a certain discussion in here, mostly because it is a mess of random arguments, public displays of mental illness, and the seeking of answers that were already provided several times over. So, instead, let's sync up.
>"Patreon?"
The importer is being fixed. It is broken for complicated reasons that are not impossible, but certainly not trivial to resolve. I promise. I believe it is the foundational core to the entire project, and will not leave it in disrepair longer than is needed.
Moreso, new archiver designs intended to replace Kitsune are being drafted, to the aim of preventing things like this from happening and making the reverse engineering process more streamlined, getting (you) more content from across the paywalled web faster.
>"Requests?"
The maintenance of most "community"-related things became the jobs of other team members months ago, but I stand by their current decisions.
>"Uploader?"
uploading =/= importing, and lack of content updates has absolutely nothing to do with the former
In general, there are multiple development-related reasons for why modtools and the fix never happened, and none of them are very important for the general public to know.
What you should know is that Kemono v3.0 is being worked on, with manual sharing and cloud drive snatching prioritized. Get used to the beta UI, by the way. Some form of it will be fully adopted soon-ish.

 No.20232

File: 1651032043721.png (912.95 KB, 730x754, koshiandoh.png)

The Patreon importer is now operational. Thank you for your patience.
Related post: >>20228

 No.22420

File: 1657900383738.png (750.91 KB, 1247x879, cc2aa2c66d85e4bcccd212bc47….png)

Cache maintenance is complete. For the next 24 hours things will be sluggish, but should ramp up in speed afterwards.



[Return][Go to top] [Catalog] [Post a Reply]
Delete Post [ ]
[ b / kemono / coomer ]