Announcement

Collapse
No announcement yet.

Folder Structures - Boring Topic tbh but I need help/advice

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Folder Structures - Boring Topic tbh but I need help/advice

    Right then. To give idea of problem u have to understand Sonicos PC. I`m running NTFS on all drives with indexing turned on atm btw.

    Example folder would be me racing folder. Its 80gb in size, and within the single root folder, are 3 large picture folders, 2 media folders, and 2 video folders. Each of these folders runs through a web pattern to many many subcategories. So for example in the crash videos folder there are about another 15 folders for different categories of racing, each with further folders for injury and fatality, each of which contains a further graphic folder for nasty stuff. Another folder of pictures, took 14 hours to verify which written to DVD backup, whereas a normal DVD time would be roughly 12 - 15 minutes. Basically the root folder spans out to roughly 200-250 total folders.

    Now, me problem is this. If I click on one of the picture folders, that each contain roughly 50,000 files, explorer literally locks solid for up to 30 - 45 seconds, before it opens the folder tree. Once its open theres no problem with accessing files and everything seems normal. This problem also affects music folders, and various other folders where there are thousands of files. Tbh its getting quite annoying. I`m fairly sure the problem is something to do with funneling links thru the mft for thousands of files, as the video folders (which have maybe 5000 files at maybe 70gb total) respond far far faster than the picture folders (which have maybe 150,000 files at maybe 15gb total).

    Now, I`ve been toying with the idea of totally restructuring the entire archive (which will take weeks, if not months) for quite a while, only this time having individual sets of root folders for each section, so sorta moving each folder in the archive up one level. So instead of like

    X:/Wheels/Pictures/F1

    it would be

    X:/Wheels - Pictures/F1

    All I want to know (esp from people who deal with servers), would be is this action likely to make any difference? Or has anyone had and cured this problem before? Cheers in adv ^_^

    #2
    Are you running XP?

    Comment


      #3
      Ja

      Comment


        #4
        No expert on this, but I'd wager it's not how the directories are arrange as you allude to in your proposed solution, but simply the sheer number of files (50k) it checks through.

        Possibly not an elegant solution, but couldn't each dir, e.g.

        X:/Wheels/Pictures/F1

        be subdivided into alphabetical directories?

        X:/Wheels/Pictures/F1/0
        X:/Wheels/Pictures/F1/a
        X:/Wheels/Pictures/F1/b
        ..
        X:/Wheels/Pictures/F1/z

        Assume the filenames are fairly random, it would possibly speed up directory listings by a factor of 26, or so.

        Maybe

        /shrug

        Comment


          #5
          Aye, it's just explorer getting its knickers in a twist over the large amount of files (i.e. get directory list, get file info, basic icon info, fill list box, etc). You'd fix it having smaller directory contents.

          I split all the music files I had into "./0", "./a", etc... can be done quickly with a bit of script to loop through number, then alphabet. You could automate the file movement too, 'sup to you really.

          Comment


            #6
            Hmm. Ok. I`ve restructured the largest of the picture folders over the last few days. Its now listed with root directory/racing series/season/round so that only about 2 of the 100 odd folders have more than 1000 files in. Thing is its STILL really slow (altho a fair bit faster).

            Any other ideas?

            I`m starting to think they haven`t invented a pc that can take what sonico chucks at it. >_<;;;

            Comment


              #7
              Other than defragging (which is gonna take ****in' ages), I'd suggest turning off indexing for those folders; Explorer could be jumping in and checkin' stuff out behind the scenes.

              I'll have a think, maybe there is something else you can do... I don't think any of the file systems Windows support are great when it comes to large archives of junk

              Comment


                #8
                So dividing it alphabetically didnt work?

                Comment


                  #9
                  Give the defrag thing a try even if it takes ages. I was having this identical problem with my last PC. The defrag fixed it.

                  Comment


                    #10
                    You could try telling windows not to do all sorts of things on the file listing too. Explorer likes to pull in all sorts of information from the files. Use list view too.

                    Or hell, try a 3rd party file browser.

                    Comment


                      #11
                      Well atm explorer is set to the most basic settings I use so it displays filename, size, type, and date created. Everything else is turned off because it REALLY didn`t like it and crashed on some folders.

                      I don`t think defrag will do **** all in this case as the same folders were moved onto a clean hard drive and that reacted the same, and the drive with these on was defrag several times in the last 6 months with little effects.

                      The folder that is the main problem is 7.5gb, has 68000 pictures, and has now gone from about 20 to 225 folders, so I don`t know what else to do. Especially since now the smaller (albeit still gigantic) 2 picture folders and a couple of others on the system are doing the same thing.

                      I thought indexing was supposed to aid just this kind of problem, rather than cause it in the first place?

                      If anyone can be arsed, I`ve uploaded a tree txt file of the entire archive so you can see exactly how its all laid out. Its 5mb in size just for that though. >_<

                      Comment


                        #12
                        A 3rd party file explorer which caches the directory listing and only updates the listing when you refresh (from adding files)?

                        No idea if such an application exists, I'm guessing now.

                        Comment


                          #13
                          No, indexing works kinda they same way as a database; it's a faster way to find information, but you need to feed it information to begin with. Explorer just goes through the directory file, looking for stuff. Defragging will at least collate some of the physical information together. Just copying it to a new hdd wouldn't guarentee a certain file structure either. Sounds like it isn't needed though and just the slow performance of explorer.

                          What spec is your PC, anything decent? Maybe it's just hdd performance or something, can't say I have the same problem with some of the information I have. Not enabled web view or anything? Nothing trying to look at more information, other than file stats (you know, that stupid media player thing where it thumbnails the stuff)? I'd probably be using the command line to work with such a large directory structure as some performance bottlenecks aren't going to go away (like I mentioned above). At least working off the cli, it'll only display the information when needed.

                          Comment


                            #14
                            I believe he is/was on dual MPs with a RAID setup. IIRC his ramblings on IRC

                            Comment


                              #15
                              3rd party explorer replacements I tried yonks ago but most of them are really buggy and crash, unless anyone knows a really decent newer one?

                              My explorer is literally set to basicness itself, with no thumbnails or anything, and no web or active stuff turned one. My pc is dual athlon mp2000, 512mb pc2100, 80gb os drive, and 4 200gb 8mb cache storage drives running through a raid card (they react the same way through the onboard ide though, or through a usb 2 port), and only a soundcard other than that.

                              I`m quite fortunate at the moment as I`ve got 2 fully built working machines (other one is a dual p3 500 compaq workstation that is going to run as a primary download machine), and enough bits left over to try linux out without worrying about ****ing it up. U guys reckon linux would be able to handle my stuff better?

                              Comment

                              Working...
                              X